-- Logs begin at Fri 2022-09-16 01:14:26 UTC, end at Fri 2022-09-16 02:24:21 UTC. -- Sep 16 01:39:26.897180 dv-cinder-849143-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Sep 16 01:39:26 2022] *** Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: nodename: dv-cinder-849143-2-iscsi Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: machine: x86_64 Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: clock source: unix Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: pcre jit disabled Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: detected number of CPU cores: 4 Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: current working directory: / Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: detected binary path: /usr/bin/uwsgi-core Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: your processes number limit is 47680 Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: your memory page size is 4096 bytes Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: detected max file descriptor number: 2048 Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: lock engine: pthread robust mutexes Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: thunder lock: enabled Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 16 01:39:26.908057 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 16 01:39:26.948353 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: Python main interpreter initialized at 0x55b44010f9c0 Sep 16 01:39:26.948353 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: python threads support enabled Sep 16 01:39:26.948353 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: your server socket listen backlog is limited to 100 connections Sep 16 01:39:26.948353 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: your mercy for graceful operations on workers is 90 seconds Sep 16 01:39:26.948701 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: mapped 403077 bytes (393 KB) for 2 cores Sep 16 01:39:26.948783 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: *** Operational MODE: preforking *** Sep 16 01:39:26.949033 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: *** uWSGI is running in multiple interpreter mode *** Sep 16 01:39:26.949033 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: spawned uWSGI master process (pid: 81412) Sep 16 01:39:26.949225 dv-cinder-849143-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 16 01:39:26.952103 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: spawned uWSGI worker 1 (pid: 81423, cores: 1) Sep 16 01:39:26.952103 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: spawned uWSGI worker 2 (pid: 81424, cores: 1) Sep 16 01:39:26.952103 dv-cinder-849143-2-iscsi devstack@keystone.service[81412]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 16 01:39:31.251287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 01:39:31.251287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: from cryptography.utils import int_from_bytes Sep 16 01:39:31.252112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 01:39:31.252112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: from cryptography.utils import int_from_bytes Sep 16 01:39:31.252538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 01:39:31.252538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: from cryptography.utils import int_from_bytes Sep 16 01:39:31.252646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 16 01:39:31.252646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: from cryptography.utils import int_from_bytes Sep 16 01:39:31.302578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 16 01:39:31.302752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 16 01:39:31.302940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] command line args: [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 16 01:39:31.303027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 16 01:39:31.303205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 16 01:39:31.303288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 16 01:39:31.303366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ================================================================================ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 16 01:39:31.303447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] command line args: [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 16 01:39:31.303612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] admin_token = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.303690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 16 01:39:31.303801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ================================================================================ {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 16 01:39:31.303884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config_dir = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.304046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.304132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] admin_token = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.304208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] config_source = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.304309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] config_dir = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.304387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] debug = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.304497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.304585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.305811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] config_source = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.305811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] debug = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.305930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.305997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_config_append = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] list_limit = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_dir = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] log_config_append = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.306903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] log_dir = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] log_file = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.307987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308741 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.308975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.309129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.309265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.309379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.309495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.309606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.309719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.309837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.310069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.310186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.310320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] publish_errors = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.310546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.42/identity {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.310670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.310784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] publish_errors = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.310901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311013 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] standard_threads = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.311955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] standard_threads = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_journal = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.312865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_json = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313030 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_stderr = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] use_journal = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] use_syslog = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] use_json = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] use_stderr = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] use_syslog = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.313940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 16 01:39:31.314162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.314966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.315084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.315202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.315823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.316049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.316173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.316292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.316413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.316527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.316812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.316942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.317894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.318978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319855 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.319980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.320933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.321952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.322986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.323899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324417 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.324876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.325992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.326898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.327861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.328995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.329928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.330861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.331033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.331662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.331998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.332928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.333943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.334993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.335913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.336863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.337881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.338954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.339885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.external = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.340911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] auth.external = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.password = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] auth.token = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.341944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] auth.password = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] auth.token = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.342963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.343874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] credential.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.344926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.345977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.346928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.347991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.348967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.349989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350870 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.350987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.351903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.352950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.353893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.354934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.355997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.356931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.357967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.358921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.359843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360417 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.360994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.361214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.361329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.361442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.361555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.361666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.361787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.361913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.362851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.363963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.364799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.365915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.366950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.367991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.368966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.369889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.370926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.371854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.372935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373695 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.373976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] role.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] role.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.374901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.375956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.376979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.377093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.377206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.377327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.377442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.377442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.377655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.377768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.378380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.378513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.378722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.378844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.378960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.379888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.380982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.381227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.381353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.381474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.381589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.381702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.381815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.381927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.382962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.383873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.384933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.385898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.386945 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.387057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.connection = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.387170 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.387544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.387665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.387778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.387890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.388990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.389906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81423) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 16 01:39:31.390693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 16 01:39:31.390693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81424) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 16 01:39:31.417780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.417912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.418054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.418135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.418209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.418284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.427021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.427241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.455548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.455672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.455803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.455882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.509841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.510408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.559345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.559512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.559649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.559731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.716652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.716874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.717093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.717309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.717496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.717606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.717725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.717805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.717942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.718156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.718369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.718579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.741316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.741574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.741671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.741773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.741915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.742850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.754214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.754340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.766912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.767138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.767314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.767499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.767585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.767723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.767805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.767931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.768039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.768165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.768248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81424) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.768424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.768778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.769045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81423) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 16 01:39:31.785813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.786019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 01:39:31.786115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.786296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 01:39:31.786376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.786539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 01:39:31.786630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.786814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.786896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 16 01:39:31.787070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.787150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.787323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.787407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.787574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.787652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.787826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.797088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.797336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.797450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.797611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.797719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.797877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.797994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.798156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.806554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.806707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.808136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.808269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.808442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.808518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.808689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.808770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.808968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.809054 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.809249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.809329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.809506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.809595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.809768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.809846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.810908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.834548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.835489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.836024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.836588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.836973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.837257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.841255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.841538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.841816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.841895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.842167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.842443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.848971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.849253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.849677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.849962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.854991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.855282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.855563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.855644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.855844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.855963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.856144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.856226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.856418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.856499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.856688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.856771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.856953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.857041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.857222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.857487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.872436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.872710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.873292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.873569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.876516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.877433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.880343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.880636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.881607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.881902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.884760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.886116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.888153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.889502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.891330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.891625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.891905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.892221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.892511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.892684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.892796 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.892981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.893265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.893677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.894135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.894419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.908710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.909516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.909797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.910196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.910748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.911037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.911303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.911562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.917033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.918293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.918556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.919807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.920097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.920371 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.921324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.921598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.923026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.923308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.923603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.923881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.924283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.924375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.924563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.924980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.925449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.925844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.935539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.935822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.936127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.936407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.936784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.936928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.937076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.937208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.937487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.937613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.937768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.938145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.938422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.938706 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.953494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.954794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.956556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.958112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.959439 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.960933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.961064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.962382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.962483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.962663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.963052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.963948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.964232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.964617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.969452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.970906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.970986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.973247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.974361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.974751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.975034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.975310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.976130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.976523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.976807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.977081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.983092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.983382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.983660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.984865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.985165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.985442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.990626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81423) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.990907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.991181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.991451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.991720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.992031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.992313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.992590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.992670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81424) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 16 01:39:31.992854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.992933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.993115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.993202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.993391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.993480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.993655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.993743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.993916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.994038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.994182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.994300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81423) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.994460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.994725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.994996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.995272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.995531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.995811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:31.996100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81424) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 16 01:39:32.029057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55b44010f9c0 pid: 81423 (default app) Sep 16 01:39:32.031265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55b44010f9c0 pid: 81424 (default app) Sep 16 01:39:32.032240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d527646-c944-46f4-8ef3-71f5a5a813eb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:32.032525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d527646-c944-46f4-8ef3-71f5a5a813eb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:32.032791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d527646-c944-46f4-8ef3-71f5a5a813eb None None] PATH_INFO: `/v3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:32.037643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1/1] 10.222.0.42 () {50 vars in 791 bytes} [Fri Sep 16 01:39:32 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:38.825221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5878c-3e49-41de-81a9-238c7629631e None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:38.825643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5878c-3e49-41de-81a9-238c7629631e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:38.825774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b5878c-3e49-41de-81a9-238c7629631e None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:38.830011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1/2] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:38 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:39:38.835761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:38.836316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:38.836316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:38.906843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: INFO dbcounter [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] Registered counter for database keystone Sep 16 01:39:38.926471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG oslo_db.sqlalchemy.engines [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81423) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 16 01:39:38.932659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writer thread running {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 16 01:39:39.124199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG passlib.handlers.bcrypt [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81423) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 16 01:39:39.124586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG passlib.handlers.bcrypt [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81423) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 16 01:39:39.182310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:39.183904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:39.310234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 2/3] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:38 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 475 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 16 01:39:39.623880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5878c-3e49-41de-81a9-238c7629631e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:39.693885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: INFO dbcounter [None req-59b5878c-3e49-41de-81a9-238c7629631e None None] Registered counter for database keystone Sep 16 01:39:39.715435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG oslo_db.sqlalchemy.engines [None req-59b5878c-3e49-41de-81a9-238c7629631e None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81424) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 16 01:39:39.722129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writer thread running {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 16 01:39:39.844612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b5878c-3e49-41de-81a9-238c7629631e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:39.849705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2903f505-e39a-4191-b5d9-92527c5f0e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:39.850322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2903f505-e39a-4191-b5d9-92527c5f0e0e admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:39.850600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2903f505-e39a-4191-b5d9-92527c5f0e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:39.850868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2903f505-e39a-4191-b5d9-92527c5f0e0e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:39.858916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2903f505-e39a-4191-b5d9-92527c5f0e0e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:39.912613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING keystone.common.rbac_enforcer.enforcer [None req-2903f505-e39a-4191-b5d9-92527c5f0e0e admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 16 01:39:39.912905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2903f505-e39a-4191-b5d9-92527c5f0e0e admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:39.922034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 2/4] 10.222.0.42 () {58 vars in 1324 bytes} [Fri Sep 16 01:39:39 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 01:39:39.928716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:39.945593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-455f6ea2-6625-45b1-a5d5-7268004bc475 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:39.949886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc55f3ee-1607-4115-8fc1-2b388cc8b5bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:39.950981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55f3ee-1607-4115-8fc1-2b388cc8b5bc admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:39.951256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55f3ee-1607-4115-8fc1-2b388cc8b5bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:39.951520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc55f3ee-1607-4115-8fc1-2b388cc8b5bc admin admin] PATH_INFO: `/v3/projects/efec9f6da9c342038dd7466c677c23ad` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:39.953099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc55f3ee-1607-4115-8fc1-2b388cc8b5bc admin admin] RBAC: Authorizing `identity:get_project(project_id=efec9f6da9c342038dd7466c677c23ad)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:40.007196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING keystone.common.rbac_enforcer.enforcer [None req-cc55f3ee-1607-4115-8fc1-2b388cc8b5bc admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 16 01:39:40.007527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc55f3ee-1607-4115-8fc1-2b388cc8b5bc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:40.011955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 3/5] 10.222.0.42 () {56 vars in 1207 bytes} [Fri Sep 16 01:39:39 2022] GET /identity/v3/projects/efec9f6da9c342038dd7466c677c23ad => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:42.151701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34667a0-9dd2-4d3d-8699-2c401e2b36ab None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:42.152227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34667a0-9dd2-4d3d-8699-2c401e2b36ab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:42.152307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34667a0-9dd2-4d3d-8699-2c401e2b36ab None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:42.156294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 3/6] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:42.161881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d98f5-6b6a-4d40-977b-46b24646bc56 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:42.162248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d98f5-6b6a-4d40-977b-46b24646bc56 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:42.162435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0d98f5-6b6a-4d40-977b-46b24646bc56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:42.182524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5d0d98f5-6b6a-4d40-977b-46b24646bc56 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:42.184081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5d0d98f5-6b6a-4d40-977b-46b24646bc56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:42.192403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 4/7] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:42 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 16 01:39:42.463067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34667a0-9dd2-4d3d-8699-2c401e2b36ab None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:42.475766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34667a0-9dd2-4d3d-8699-2c401e2b36ab None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:42.480062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7cd607-bfa8-4f4a-a5b3-2148ff4aa849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:42.485135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7cd607-bfa8-4f4a-a5b3-2148ff4aa849 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:42.485135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7cd607-bfa8-4f4a-a5b3-2148ff4aa849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:42.485135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7cd607-bfa8-4f4a-a5b3-2148ff4aa849 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:42.489555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7cd607-bfa8-4f4a-a5b3-2148ff4aa849 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:42.491299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7cd607-bfa8-4f4a-a5b3-2148ff4aa849 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:42.499837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 4/8] 10.222.0.42 () {58 vars in 1324 bytes} [Fri Sep 16 01:39:42 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:39:42.505368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d98f5-6b6a-4d40-977b-46b24646bc56 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:42.514609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0d98f5-6b6a-4d40-977b-46b24646bc56 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:42.518951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c677a4-fe50-43ef-9b5c-6924806ce582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:42.519562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c677a4-fe50-43ef-9b5c-6924806ce582 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:42.519844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c677a4-fe50-43ef-9b5c-6924806ce582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:42.520142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c677a4-fe50-43ef-9b5c-6924806ce582 admin admin] PATH_INFO: `/v3/users/a00246e1b29a4746b4e48d4dc1f5618c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:42.521444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c677a4-fe50-43ef-9b5c-6924806ce582 admin admin] RBAC: Authorizing `identity:get_user(user_id=a00246e1b29a4746b4e48d4dc1f5618c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:42.523274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c677a4-fe50-43ef-9b5c-6924806ce582 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:42.524811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 5/9] 10.222.0.42 () {56 vars in 1198 bytes} [Fri Sep 16 01:39:42 2022] GET /identity/v3/users/a00246e1b29a4746b4e48d4dc1f5618c => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:44.745405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9f345b-097c-44e4-9445-66a863a3278b None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:44.745968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9f345b-097c-44e4-9445-66a863a3278b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:44.745968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9f345b-097c-44e4-9445-66a863a3278b None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:44.747249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 5/10] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:44.753166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f007a5-03f8-41a1-92ff-601a8d30e988 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:44.753567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f007a5-03f8-41a1-92ff-601a8d30e988 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:44.753777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f007a5-03f8-41a1-92ff-601a8d30e988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:44.767566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d74b303-5f09-4c63-834c-90b910e8263c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:44.767955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d74b303-5f09-4c63-834c-90b910e8263c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:44.768402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d74b303-5f09-4c63-834c-90b910e8263c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:44.769923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 6/11] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:39:44.772105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-18f007a5-03f8-41a1-92ff-601a8d30e988 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:44.773290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-18f007a5-03f8-41a1-92ff-601a8d30e988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:44.774919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:44.775190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:44.775454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:44.818347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG passlib.handlers.bcrypt [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81424) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 16 01:39:44.818727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG passlib.handlers.bcrypt [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81424) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 16 01:39:44.856374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 6/12] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:39:44.867913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:44.869505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:44.968739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 7/13] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:39:45.122673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f007a5-03f8-41a1-92ff-601a8d30e988 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.147541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:45.232912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f007a5-03f8-41a1-92ff-601a8d30e988 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.235740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.267000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.267631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.268471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.268755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.280175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:39:45.289283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:39:45.290240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:39:45.291669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 7/14] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:39:45.296162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.337810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13a8aa1-3b31-49a4-ba1e-d8ce6096c946 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.340103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:45.376220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.377005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.377800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.378069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.379522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:45.398434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f754c6-69d0-4e08-9046-708c59f2c820 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.430796 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.431390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.431663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.431947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.432607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:39:45.432901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:45.434383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:45.446241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:45.446562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 8/15] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:39:45.447763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:45.452336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.544992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 8/16] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:39:45.550005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f8513f-09b3-4757-92d5-ec471799f87b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.551113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.554245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:45.584913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.585542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.585844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.586117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] PATH_INFO: `/v3/users/a00246e1b29a4746b4e48d4dc1f5618c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.587257 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] RBAC: Authorizing `identity:get_user(user_id=a00246e1b29a4746b4e48d4dc1f5618c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:45.588734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:45.590294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 9/17] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/users/a00246e1b29a4746b4e48d4dc1f5618c => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:39:45.595627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.610017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:45.650646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6baa0f4a-0c3d-4fbf-b767-1a1ca225208f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.691237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.692118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.692388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.692656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.695152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649342a6-83f4-4282-983b-49bdcc6b2781 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.711424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:45.712906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:45.730253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.730877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.731155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.731424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:39:45.732162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:39:45.733024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:39:45.733953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:39:45.733953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:39:45.733953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:39:45.734113 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:45.734176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 9/18] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:39:45.734236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:45.735486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 10/19] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:39:45.739045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.741062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:45.797681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:45.824306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:45.844212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f77517c-4bc1-4bf0-a050-bbacb576444a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.844765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660a4e58-cfc0-440d-8732-a61039375205 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:45.879534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a9ebb-f69f-47fe-9296-9a5339df5a67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.880175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a9ebb-f69f-47fe-9296-9a5339df5a67 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.880279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54845da7-c982-45cc-b6cf-93f0c0dc7ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:45.880453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a9ebb-f69f-47fe-9296-9a5339df5a67 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.880720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a9ebb-f69f-47fe-9296-9a5339df5a67 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.880870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54845da7-c982-45cc-b6cf-93f0c0dc7ad8 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:45.881149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54845da7-c982-45cc-b6cf-93f0c0dc7ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:45.881418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54845da7-c982-45cc-b6cf-93f0c0dc7ad8 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:45.881506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a9ebb-f69f-47fe-9296-9a5339df5a67 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.domain.id=default, user.id=a00246e1b29a4746b4e48d4dc1f5618c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:39:45.881738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a9ebb-f69f-47fe-9296-9a5339df5a67 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:45.882059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54845da7-c982-45cc-b6cf-93f0c0dc7ad8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:39:45.882357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54845da7-c982-45cc-b6cf-93f0c0dc7ad8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:45.883176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a9ebb-f69f-47fe-9296-9a5339df5a67 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:45.883722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54845da7-c982-45cc-b6cf-93f0c0dc7ad8 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:45.890071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 11/20] 10.222.0.42 () {56 vars in 1322 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/role_assignments?user.id=a00246e1b29a4746b4e48d4dc1f5618c&scope.domain.id=default&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 233 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:45.902033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 10/21] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:39:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:39:48.352382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944b5847-1392-4eb4-9b16-b956ef92d484 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:48.352382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944b5847-1392-4eb4-9b16-b956ef92d484 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:48.352382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944b5847-1392-4eb4-9b16-b956ef92d484 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:48.353554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 12/22] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:48.357565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45155b3d-b193-4755-bb62-8c119283a522 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:48.358345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45155b3d-b193-4755-bb62-8c119283a522 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:48.358836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45155b3d-b193-4755-bb62-8c119283a522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:48.389861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-45155b3d-b193-4755-bb62-8c119283a522 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:48.391406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-45155b3d-b193-4755-bb62-8c119283a522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:48.482412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 11/23] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:39:48.748481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d066196b-2988-4652-af93-9e749aadeb16 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:48.749146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d066196b-2988-4652-af93-9e749aadeb16 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:48.749652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d066196b-2988-4652-af93-9e749aadeb16 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:48.751459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 13/24] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:48.756897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd44cfd-3a38-4e00-b43a-ea186fd2d43b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:48.757188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd44cfd-3a38-4e00-b43a-ea186fd2d43b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:48.757459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd44cfd-3a38-4e00-b43a-ea186fd2d43b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:48.780375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-cfd44cfd-3a38-4e00-b43a-ea186fd2d43b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:48.781904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cfd44cfd-3a38-4e00-b43a-ea186fd2d43b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:48.783173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a6c0d-8e3e-4c4c-bbc4-353e392c5fda None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:48.783728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a6c0d-8e3e-4c4c-bbc4-353e392c5fda None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:48.784495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a6c0d-8e3e-4c4c-bbc4-353e392c5fda None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:48.786508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 14/25] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:48.791324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246ea4d-de23-4c63-8d9e-b251aaee21a9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:48.791851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246ea4d-de23-4c63-8d9e-b251aaee21a9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:48.792360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d246ea4d-de23-4c63-8d9e-b251aaee21a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:48.828941 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d246ea4d-de23-4c63-8d9e-b251aaee21a9 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:48.830685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d246ea4d-de23-4c63-8d9e-b251aaee21a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:48.850504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:48.886659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 12/26] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:39:48.934339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd44cfd-3a38-4e00-b43a-ea186fd2d43b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:48.986684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 15/27] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:39:49.059810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd44cfd-3a38-4e00-b43a-ea186fd2d43b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:49.103394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:49.104346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.104768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.105197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.118274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:39:49.133311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:39:49.134393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:39:49.135252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:39:49.137718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 13/28] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:39:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:39:49.143879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d246ea4d-de23-4c63-8d9e-b251aaee21a9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:49.237931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:49.266547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d246ea4d-de23-4c63-8d9e-b251aaee21a9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:49.301842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174dc68-0414-443b-988a-27422420af45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:49.302709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174dc68-0414-443b-988a-27422420af45 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.303139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174dc68-0414-443b-988a-27422420af45 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.303567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174dc68-0414-443b-988a-27422420af45 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.304396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174dc68-0414-443b-988a-27422420af45 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:39:49.304835 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174dc68-0414-443b-988a-27422420af45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:49.306370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174dc68-0414-443b-988a-27422420af45 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:49.319156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 16/29] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:39:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:49.324890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:49.336095 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174dc68-0414-443b-988a-27422420af45 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:49.397719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:49.454831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174dc68-0414-443b-988a-27422420af45 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:49.490849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:49.491999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.492379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.492744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.493525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:49.494644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419488bf-cb4a-42ce-bb32-df4b31d4bedf None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:49.495078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:49.501472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:49.532346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1ede3d-8a5d-4058-92a5-890a55d5f836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:49.533606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1ede3d-8a5d-4058-92a5-890a55d5f836 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.534058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1ede3d-8a5d-4058-92a5-890a55d5f836 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.534544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1ede3d-8a5d-4058-92a5-890a55d5f836 None admin] PATH_INFO: `/v3/users/a00246e1b29a4746b4e48d4dc1f5618c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.536227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1ede3d-8a5d-4058-92a5-890a55d5f836 None admin] RBAC: Authorizing `identity:get_user(user_id=a00246e1b29a4746b4e48d4dc1f5618c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:49.537893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1ede3d-8a5d-4058-92a5-890a55d5f836 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:49.539684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 14/30] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:39:49 2022] GET /identity/v3/users/a00246e1b29a4746b4e48d4dc1f5618c => generated 247 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:49.541056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84cce59-df15-4070-9fb0-52f2736326ae None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.541591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84cce59-df15-4070-9fb0-52f2736326ae None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.542033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84cce59-df15-4070-9fb0-52f2736326ae None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.543679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 15/31] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:49.544655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84cce59-df15-4070-9fb0-52f2736326ae None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:49.636477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 17/32] 10.222.0.42 () {60 vars in 1131 bytes} [Fri Sep 16 01:39:49 2022] POST /identity/v3/roles => generated 219 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:39:49.638065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:49.655005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84cce59-df15-4070-9fb0-52f2736326ae None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:49.658121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:49.694045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98a97ce-0fe0-41c5-9ac7-cf658bf972b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:49.694585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a97ce-0fe0-41c5-9ac7-cf658bf972b5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.694866 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a97ce-0fe0-41c5-9ac7-cf658bf972b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.695136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a97ce-0fe0-41c5-9ac7-cf658bf972b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.708247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98a97ce-0fe0-41c5-9ac7-cf658bf972b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:49.709605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98a97ce-0fe0-41c5-9ac7-cf658bf972b5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-b98a97ce-0fe0-41c5-9ac7-cf658bf972b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:39:49.725156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:39:49.726050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:39:49.731167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:39:49.731167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:39:49.731167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:39:49.731167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 16/33] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:39:49.731167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23ae7a-b278-436a-994f-f2387b144483 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.731167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23ae7a-b278-436a-994f-f2387b144483 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.731167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c23ae7a-b278-436a-994f-f2387b144483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.748728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8c23ae7a-b278-436a-994f-f2387b144483 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:49.749965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8c23ae7a-b278-436a-994f-f2387b144483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:49.773188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a297cbe-23bc-4f1d-9dee-4c1dc7c435c5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:49.819485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:49.820530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.821003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.821513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.823125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:49.825185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:49.826775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 17/34] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:39:49.827342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 18/35] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:49.827688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c23ae7a-b278-436a-994f-f2387b144483 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:49.834713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:49.922153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:49.940005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c23ae7a-b278-436a-994f-f2387b144483 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:49.944580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:49.982225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:49.982862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:49.983138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:49.983406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:49.984093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:39:49.984397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:49.985812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:50.011305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 18/36] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:39:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:50.012524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea936c55-4fd8-4c12-aeb1-ebbe04c4ee89 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:50.020969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:50.054200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:50.055217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:50.055713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:50.056217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] PATH_INFO: `/v3/domains/default/users/a00246e1b29a4746b4e48d4dc1f5618c/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:50.058447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=a00246e1b29a4746b4e48d4dc1f5618c, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:50.060115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:50.081699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 19/37] 10.222.0.42 () {58 vars in 1361 bytes} [Fri Sep 16 01:39:49 2022] PUT /identity/v3/domains/default/users/a00246e1b29a4746b4e48d4dc1f5618c/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:39:50.144572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759ae647-373c-42ec-9981-c5c7e603c0a2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:50.180876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f46f73e-9f28-4279-ad35-759d160831ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:50.181423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f46f73e-9f28-4279-ad35-759d160831ca None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:50.181703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f46f73e-9f28-4279-ad35-759d160831ca None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:50.181972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f46f73e-9f28-4279-ad35-759d160831ca None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:50.182640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f46f73e-9f28-4279-ad35-759d160831ca None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:50.184164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f46f73e-9f28-4279-ad35-759d160831ca None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:50.212107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:50.331802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 19/38] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:39:50 2022] POST /identity/v3/projects => generated 301 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:39:50.408077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:50.437977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0a309c-cdec-482d-a283-7b1975c2d088 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:50.438444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0a309c-cdec-482d-a283-7b1975c2d088 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:50.438444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0a309c-cdec-482d-a283-7b1975c2d088 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:50.439893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 20/39] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:39:50.445599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607023a-c013-4d55-988f-a80caa380fdb None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:50.445947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607023a-c013-4d55-988f-a80caa380fdb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:50.446231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607023a-c013-4d55-988f-a80caa380fdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:50.474235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b607023a-c013-4d55-988f-a80caa380fdb None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:50.476605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b607023a-c013-4d55-988f-a80caa380fdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:50.555247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb30fd-d824-4c7b-a739-4f4a24703b56 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:50.563679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 21/40] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:39:50.565838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06ac13-b9bd-4d94-a709-b5ffebd93cf2 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:50.566076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06ac13-b9bd-4d94-a709-b5ffebd93cf2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:50.566275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06ac13-b9bd-4d94-a709-b5ffebd93cf2 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:50.567573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 22/41] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:50.577488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799560cc-c61b-4b98-8797-7a9a426b4a7a None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:50.577758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799560cc-c61b-4b98-8797-7a9a426b4a7a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:50.578038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799560cc-c61b-4b98-8797-7a9a426b4a7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:50.587228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:50.606021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-799560cc-c61b-4b98-8797-7a9a426b4a7a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:50.609289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-799560cc-c61b-4b98-8797-7a9a426b4a7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:50.610677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:50.611318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:50.612162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:50.612457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:50.613123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:50.614529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:50.634296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 20/42] 10.222.0.42 () {60 vars in 1131 bytes} [Fri Sep 16 01:39:50 2022] POST /identity/v3/roles => generated 225 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:39:50.698309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 23/43] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:39:50.875191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:50.979385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799560cc-c61b-4b98-8797-7a9a426b4a7a None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:50.982346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42771eb-9204-4833-a4f2-bb9c5906fb70 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:51.018108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:51.018747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:51.019027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:51.019334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:51.020457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:51.021913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:51.023475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 21/44] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:51.029326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:51.047078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:51.085935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:51.100570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799560cc-c61b-4b98-8797-7a9a426b4a7a None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:51.138679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f638a8-f504-4be5-8ed3-e66654fc5945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:51.139298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f638a8-f504-4be5-8ed3-e66654fc5945 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:51.139423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae36e3b-6832-4a64-b072-b4375fd72feb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:51.139584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f638a8-f504-4be5-8ed3-e66654fc5945 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:51.139852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f638a8-f504-4be5-8ed3-e66654fc5945 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:51.140558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f638a8-f504-4be5-8ed3-e66654fc5945 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:51.142384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f638a8-f504-4be5-8ed3-e66654fc5945 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:51.160917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 24/45] 10.222.0.42 () {60 vars in 1131 bytes} [Fri Sep 16 01:39:50 2022] POST /identity/v3/roles => generated 223 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:39:51.182461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348926e4-8fef-4999-8f8a-60c2f4cb6473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:51.183093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348926e4-8fef-4999-8f8a-60c2f4cb6473 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:51.183415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348926e4-8fef-4999-8f8a-60c2f4cb6473 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:51.183691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348926e4-8fef-4999-8f8a-60c2f4cb6473 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:51.184420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348926e4-8fef-4999-8f8a-60c2f4cb6473 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:51.185916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348926e4-8fef-4999-8f8a-60c2f4cb6473 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:51.219411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 22/46] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:39:51 2022] POST /identity/v3/projects => generated 312 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:39:52.562143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3bbc3c-4265-4e6e-b259-8fa154bac217 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:52.562660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3bbc3c-4265-4e6e-b259-8fa154bac217 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:52.562811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3bbc3c-4265-4e6e-b259-8fa154bac217 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:52.564200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 25/47] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:39:52.570543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef3e82-d107-41b4-8cd8-9eb9a55e0aaa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:52.570948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef3e82-d107-41b4-8cd8-9eb9a55e0aaa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:52.571220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef3e82-d107-41b4-8cd8-9eb9a55e0aaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:52.590620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c5ef3e82-d107-41b4-8cd8-9eb9a55e0aaa None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:52.592141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c5ef3e82-d107-41b4-8cd8-9eb9a55e0aaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:52.763199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 23/48] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:39:53.029460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3bbc3c-4265-4e6e-b259-8fa154bac217 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:53.131334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3bbc3c-4265-4e6e-b259-8fa154bac217 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:53.149380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:53.167539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:53.168194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.168474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.168742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.179303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:39:53.184113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:39:53.185070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:39:53.185848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 26/49] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:39:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:39:53.191201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef3e82-d107-41b4-8cd8-9eb9a55e0aaa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:53.224653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:53.293053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef3e82-d107-41b4-8cd8-9eb9a55e0aaa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:53.328965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175a6507-d2c2-4488-8e5d-0eb2ceed17fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:53.329573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a6507-d2c2-4488-8e5d-0eb2ceed17fe None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.329856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a6507-d2c2-4488-8e5d-0eb2ceed17fe None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.330157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175a6507-d2c2-4488-8e5d-0eb2ceed17fe None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.330966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175a6507-d2c2-4488-8e5d-0eb2ceed17fe None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:39:53.331270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175a6507-d2c2-4488-8e5d-0eb2ceed17fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:53.332683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175a6507-d2c2-4488-8e5d-0eb2ceed17fe None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:53.340814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 24/50] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:39:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:53.346415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:53.402908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39412ab9-66c7-4f88-a1b6-eded6c550d2a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.403198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39412ab9-66c7-4f88-a1b6-eded6c550d2a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.403480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39412ab9-66c7-4f88-a1b6-eded6c550d2a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.404855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 25/51] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:39:53.409818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdbc260-228f-41ea-bc9a-042afcd2dcab None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.410104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdbc260-228f-41ea-bc9a-042afcd2dcab None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.410383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdbc260-228f-41ea-bc9a-042afcd2dcab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.428713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2fdbc260-228f-41ea-bc9a-042afcd2dcab None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:53.429900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2fdbc260-228f-41ea-bc9a-042afcd2dcab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:53.450945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72659d7-8a00-4060-ad4e-b1129fb2eebd None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:53.484066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:53.484686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.484968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.485239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] PATH_INFO: `/v3/users/a00246e1b29a4746b4e48d4dc1f5618c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.486879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] RBAC: Authorizing `identity:get_user(user_id=a00246e1b29a4746b4e48d4dc1f5618c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:53.488355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:53.489917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 27/52] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:39:53 2022] GET /identity/v3/users/a00246e1b29a4746b4e48d4dc1f5618c => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:53.495395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:53.496312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 26/53] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:39:53.643277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:53.694965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c473b946-df53-46a5-bf9e-4d9ec6870ce2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:53.728265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:53.728892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.729174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.729535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.730557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:53.732020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:53.733594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 28/54] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:39:53.740390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdbc260-228f-41ea-bc9a-042afcd2dcab None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:53.765793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:53.767191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:53.847781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdbc260-228f-41ea-bc9a-042afcd2dcab None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:53.868253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394d4dcd-fc83-4860-97da-2f29517d5b4c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:53.881126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:53.881933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.882171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:53.882514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.882785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.883601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.domain.id=default, user.id=a00246e1b29a4746b4e48d4dc1f5618c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:39:53.884008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:53.885522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:53.891592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 27/55] 10.222.0.42 () {56 vars in 1322 bytes} [Fri Sep 16 01:39:53 2022] GET /identity/v3/role_assignments?user.id=a00246e1b29a4746b4e48d4dc1f5618c&scope.domain.id=default&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 530 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:53.903598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8199c78f-64fd-49f3-a826-0e16a351064a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:53.904184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8199c78f-64fd-49f3-a826-0e16a351064a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:53.904471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8199c78f-64fd-49f3-a826-0e16a351064a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:53.904747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8199c78f-64fd-49f3-a826-0e16a351064a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:53.905949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8199c78f-64fd-49f3-a826-0e16a351064a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:53.907440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8199c78f-64fd-49f3-a826-0e16a351064a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:53.909542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 29/56] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:39:53.915209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:54.012888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:54.024656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ff5f8e-280e-4ec4-959f-f7a7fcdb4b3e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:54.059664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f8e120-7b34-4e96-bb23-bb7939dec7f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:54.060305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f8e120-7b34-4e96-bb23-bb7939dec7f0 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:54.060577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f8e120-7b34-4e96-bb23-bb7939dec7f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:54.060847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f8e120-7b34-4e96-bb23-bb7939dec7f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:54.061510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f8e120-7b34-4e96-bb23-bb7939dec7f0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:54.063368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f8e120-7b34-4e96-bb23-bb7939dec7f0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:54.090833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 28/57] 10.222.0.42 () {60 vars in 1140 bytes} [Fri Sep 16 01:39:53 2022] POST /identity/v3/projects => generated 298 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:39:56.233272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b59e215-e8f1-44d4-bd71-efbf24a8ac0f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:56.233806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b59e215-e8f1-44d4-bd71-efbf24a8ac0f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:56.233806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b59e215-e8f1-44d4-bd71-efbf24a8ac0f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:56.235171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 30/58] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:39:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:39:56.240890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4280230e-0dbb-4d80-a6e3-0afdae849abc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:56.241302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4280230e-0dbb-4d80-a6e3-0afdae849abc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:56.241455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4280230e-0dbb-4d80-a6e3-0afdae849abc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:56.259622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4280230e-0dbb-4d80-a6e3-0afdae849abc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:39:56.260834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4280230e-0dbb-4d80-a6e3-0afdae849abc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:39:56.326408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 29/59] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:39:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:39:56.597969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b59e215-e8f1-44d4-bd71-efbf24a8ac0f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:56.697398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b59e215-e8f1-44d4-bd71-efbf24a8ac0f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:56.729973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dc540d-ba5e-4a7a-9782-4eff7ee33f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:56.730594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc540d-ba5e-4a7a-9782-4eff7ee33f87 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:56.730877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc540d-ba5e-4a7a-9782-4eff7ee33f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:56.731152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dc540d-ba5e-4a7a-9782-4eff7ee33f87 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:56.732276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dc540d-ba5e-4a7a-9782-4eff7ee33f87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:56.733711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dc540d-ba5e-4a7a-9782-4eff7ee33f87 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:56.735253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 31/60] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:39:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:39:56.741401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4280230e-0dbb-4d80-a6e3-0afdae849abc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:39:56.809950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:39:56.839906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4280230e-0dbb-4d80-a6e3-0afdae849abc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:39:56.873232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ad9b1f-5ac6-475f-8581-5cf4a83a633a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:39:56.873828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad9b1f-5ac6-475f-8581-5cf4a83a633a None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:39:56.874111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad9b1f-5ac6-475f-8581-5cf4a83a633a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:39:56.874385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ad9b1f-5ac6-475f-8581-5cf4a83a633a None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:39:56.875045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ad9b1f-5ac6-475f-8581-5cf4a83a633a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:39:56.876528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ad9b1f-5ac6-475f-8581-5cf4a83a633a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:39:56.903285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 30/61] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:39:56 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:00.245298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc82569-4fa3-43e8-9f4a-9a969856ca28 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.246361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc82569-4fa3-43e8-9f4a-9a969856ca28 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.246874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc82569-4fa3-43e8-9f4a-9a969856ca28 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.248637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 32/62] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:00.256140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cadf6b-19b1-4a7d-bc4a-46f3fe3c18af None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.257082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cadf6b-19b1-4a7d-bc4a-46f3fe3c18af None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.257656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cadf6b-19b1-4a7d-bc4a-46f3fe3c18af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.285133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d6cadf6b-19b1-4a7d-bc4a-46f3fe3c18af None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:00.286816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d6cadf6b-19b1-4a7d-bc4a-46f3fe3c18af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:00.375042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa409c68-6d39-4e12-afa8-e96bf0332897 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.375679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa409c68-6d39-4e12-afa8-e96bf0332897 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.376180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa409c68-6d39-4e12-afa8-e96bf0332897 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.378590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 33/63] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:00.390361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 31/64] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:00.392788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647a6cf-0564-436d-9691-5837dae22d3e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.393347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647a6cf-0564-436d-9691-5837dae22d3e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.393986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647a6cf-0564-436d-9691-5837dae22d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.416634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de3cfba-8020-4d55-89ac-e8b68ce2b271 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.416929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de3cfba-8020-4d55-89ac-e8b68ce2b271 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.417186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de3cfba-8020-4d55-89ac-e8b68ce2b271 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.418512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 32/65] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:00.424278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2394a-e382-4bde-bf73-8c620223d6d7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.424714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2394a-e382-4bde-bf73-8c620223d6d7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.424997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e2394a-e382-4bde-bf73-8c620223d6d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.428894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4647a6cf-0564-436d-9691-5837dae22d3e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:00.430104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4647a6cf-0564-436d-9691-5837dae22d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:00.461159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-36e2394a-e382-4bde-bf73-8c620223d6d7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:00.462350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-36e2394a-e382-4bde-bf73-8c620223d6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:00.527262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:00.566612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 34/66] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:00.586498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:00.605550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 33/67] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:00.699012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647a6cf-0564-436d-9691-5837dae22d3e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:00.820866 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647a6cf-0564-436d-9691-5837dae22d3e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:00.848333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa65a40e-f0dd-4e89-a549-2a5574c36b1f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.848997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa65a40e-f0dd-4e89-a549-2a5574c36b1f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.849497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa65a40e-f0dd-4e89-a549-2a5574c36b1f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.852129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 34/68] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:00.859479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca040c35-faec-47d6-bde9-ac7573af5276 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.860131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca040c35-faec-47d6-bde9-ac7573af5276 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.860756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca040c35-faec-47d6-bde9-ac7573af5276 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.866993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:00.867651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:00.867927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:00.868234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:00.880612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:00.885824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:00.886669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:00.887669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:00.889425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 35/69] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:00.897488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:00.899337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ca040c35-faec-47d6-bde9-ac7573af5276 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:00.900999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ca040c35-faec-47d6-bde9-ac7573af5276 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:00.966459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.008805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 35/70] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:01.008805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca040c35-faec-47d6-bde9-ac7573af5276 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.031332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e610405-05d6-4b6f-9aec-53e817d0902a None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.070300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.070833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.071115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.071385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.072180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:01.072602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:01.073978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:01.090816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 36/71] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:01.090816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.114360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca040c35-faec-47d6-bde9-ac7573af5276 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.132945 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.150803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634f3404-90fe-40c0-9088-c42c7739c851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.151450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f3404-90fe-40c0-9088-c42c7739c851 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.151719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f3404-90fe-40c0-9088-c42c7739c851 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.152024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f3404-90fe-40c0-9088-c42c7739c851 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.162921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634f3404-90fe-40c0-9088-c42c7739c851 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-634f3404-90fe-40c0-9088-c42c7739c851 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:01.167833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:01.168859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:01.169810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:01.173852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 36/72] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:01.176480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036d817-0187-439f-83e3-f46eb4cf30d3 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.176773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036d817-0187-439f-83e3-f46eb4cf30d3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.177050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036d817-0187-439f-83e3-f46eb4cf30d3 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.178340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 37/73] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:01.179049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a036d817-0187-439f-83e3-f46eb4cf30d3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.210631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811f1216-8850-41ae-8b0c-5cdc73c065ef None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.212183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.248717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.249359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.249624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.249892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.261250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:01.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:01.267290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:01.268072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 37/74] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:01.268691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.291808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a036d817-0187-439f-83e3-f46eb4cf30d3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.325736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41446037-195d-481b-82b6-a1c26d5557d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.326290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41446037-195d-481b-82b6-a1c26d5557d2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.326573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41446037-195d-481b-82b6-a1c26d5557d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.326846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41446037-195d-481b-82b6-a1c26d5557d2 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.341122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41446037-195d-481b-82b6-a1c26d5557d2 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:01.342874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41446037-195d-481b-82b6-a1c26d5557d2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:01.345023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 38/75] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:01.347229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9896a7-7f34-4f53-b50c-2212445785f3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.347516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9896a7-7f34-4f53-b50c-2212445785f3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.347780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9896a7-7f34-4f53-b50c-2212445785f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.365600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bc9896a7-7f34-4f53-b50c-2212445785f3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:01.366793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bc9896a7-7f34-4f53-b50c-2212445785f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:01.369680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c4237b-5e23-4c42-a3c3-c44734208b75 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.395397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.404474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.405077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.405355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.405631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.406291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:01.406591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:01.408822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:01.416788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 38/76] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:01.417609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.436247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 39/77] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:01.436947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9896a7-7f34-4f53-b50c-2212445785f3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.447887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.526334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c6cac-d527-4a6b-8dff-aa233bf4774a None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.550496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9896a7-7f34-4f53-b50c-2212445785f3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.561697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.562455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.562738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.563010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.563838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:01.564212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:01.565959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:01.574099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 39/78] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:01.574716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.589316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.589882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.590160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.590430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.602531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:01.607379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:01.608255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:01.609021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 40/79] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:01.610243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.641076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.669147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.685591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3235708-6fc9-4d57-b969-fbd8c0cd6639 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.717849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccdeeb4-76ec-4f7a-82a2-f3b0c6cafacc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.723674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.724307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.724584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.724850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.751986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.753027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.753303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.753575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] PATH_INFO: `/v3/users/a00246e1b29a4746b4e48d4dc1f5618c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.754713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] RBAC: Authorizing `identity:get_user(user_id=a00246e1b29a4746b4e48d4dc1f5618c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:01.756205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:01.757767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 41/80] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/users/a00246e1b29a4746b4e48d4dc1f5618c => generated 247 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:01.759215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.790807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:01.792314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:01.853253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.859993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c6efe4-8829-476b-acdf-23878a084a83 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.876028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:01.891917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 40/81] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:01.893088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.895344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:01.896005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:01.896284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:01.896761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:01.897699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:01.899138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:01.900715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 42/82] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:01.901461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:01.992963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866eba7b-02e0-4cb1-9269-a08033b2b7ac None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:01.999051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95c7b7f-9413-4d68-b154-4f7b49096f19 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.026863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.027459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.027747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.028043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.028700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:02.028997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.030364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.033586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.034209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.034490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.034767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.035883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.037326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.038882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 43/83] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:02.039651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.040070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 41/84] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:02.041049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.073296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.089841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.141733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe03a88-9b20-4a17-a488-7931946888fb None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.142928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8eb93f-bfd5-4ee9-b8d0-6d31f4a86d4f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.177173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.177370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.177799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.177973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.178080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.178256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.178339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.178532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.179451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.180925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.182546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 42/85] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:02.183311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.246084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.247547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.276577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.280793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998d2072-06ce-467c-92dd-b3d1817a17f1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.296575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.315072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.315676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.315978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.316257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.346838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 44/86] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:02.347752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.383292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.384784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.444267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087732aa-52e8-41da-82df-778ffeb950a3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.472353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.478322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.478930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.479210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.479489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.480166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.481603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.487460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 43/87] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:02.488342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.504807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 45/88] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:02 2022] POST /identity/v3/users => generated 289 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:02.505628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.521951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.618040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec0a95-08d2-47d4-83cb-89aed512a86e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.618386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37a46d2-6d03-4ccb-a2c0-9cc7e39e7bc4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.656404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.657398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.657398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.657558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.658768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:02.659154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.660593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.665677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.666327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.666604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.672293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.672458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 46/89] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/role_assignments?user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:02.672458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.675401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.675401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.677906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 44/90] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:02.678692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.746752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.821797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:02.837802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4c45c3-1292-4920-867c-aa1fb7d93a0b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.849220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fab5ccf-de1a-4a88-9f03-3b2e4410ca81 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:02.884540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.885527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.885985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.886459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.888190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.889872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.891721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 47/91] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:02.894483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:02.898165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:02.898818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:02.899098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:02.899366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:02.901962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:02.903406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:02.905329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 45/92] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:02.905971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:03.038857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0606a93-b75d-47b1-b753-654a1c1bf8d3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:03.045040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ae0ff5-f0c2-40eb-84aa-92335be099f6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:03.069994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:03.084499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:03.085130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:03.085409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:03.085681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:03.086713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=a00246e1b29a4746b4e48d4dc1f5618c) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:03.087413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:03.089177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:03.092027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:03.092652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:03.092934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:03.093209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:03.098158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 46/93] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/role_assignments?user.id=a00246e1b29a4746b4e48d4dc1f5618c&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:03.098994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:03.138931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:03.236756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:03.243970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:03.291465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76234a4-c46e-4f0c-b424-528d3225203e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:03.363132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:03.364140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:03.365584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:03.366184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:03.371967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] RBAC: Adding query filter params (role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:03.374636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:03.374636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:03.389196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 47/94] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:03 2022] GET /identity/v3/role_assignments?user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=9d68999390534c65b1cfc53d421f6412 => generated 259 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:03.416828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 48/95] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:03.422687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:03.514528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:03.537246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26efdeb-9731-4f78-90a2-3a701c6b77db None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:03.580894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba97607-ea05-4a34-a390-8a99330cb480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:03.581462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba97607-ea05-4a34-a390-8a99330cb480 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:03.581793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba97607-ea05-4a34-a390-8a99330cb480 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:03.582073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba97607-ea05-4a34-a390-8a99330cb480 None admin] PATH_INFO: `/v3/projects/94593fc673d64bf6ba670d636a646ce8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:03.583386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba97607-ea05-4a34-a390-8a99330cb480 None admin] RBAC: Authorizing `identity:get_project(project_id=94593fc673d64bf6ba670d636a646ce8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:03.584890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba97607-ea05-4a34-a390-8a99330cb480 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:03.586736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 48/96] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:03 2022] GET /identity/v3/projects/94593fc673d64bf6ba670d636a646ce8 => generated 312 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:03.592764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:03.613681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:03.718082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9a858a-1903-4175-bce8-e9f3dbaa1218 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:03.761154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe6c4ba-1617-4f9e-920a-50bd682b2486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:03.761763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe6c4ba-1617-4f9e-920a-50bd682b2486 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:03.762302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe6c4ba-1617-4f9e-920a-50bd682b2486 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:03.762577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe6c4ba-1617-4f9e-920a-50bd682b2486 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:03.763311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe6c4ba-1617-4f9e-920a-50bd682b2486 None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=94593fc673d64bf6ba670d636a646ce8, user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:03.763619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe6c4ba-1617-4f9e-920a-50bd682b2486 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:03.765114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe6c4ba-1617-4f9e-920a-50bd682b2486 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:03.771824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 49/97] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:03 2022] GET /identity/v3/role_assignments?user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797&scope.project.id=94593fc673d64bf6ba670d636a646ce8&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:05.859470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa7390b-c8ba-4eb2-a224-1f54157a3506 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:05.860037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa7390b-c8ba-4eb2-a224-1f54157a3506 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:05.860037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa7390b-c8ba-4eb2-a224-1f54157a3506 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:05.861958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 49/98] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:05.869679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f649e-6426-46c6-9850-d5f99fc952fb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:05.870603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f649e-6426-46c6-9850-d5f99fc952fb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:05.871077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3f649e-6426-46c6-9850-d5f99fc952fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:05.899418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-af3f649e-6426-46c6-9850-d5f99fc952fb None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:05.901011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-af3f649e-6426-46c6-9850-d5f99fc952fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:05.934513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a8983b-f3ee-4659-8e10-00ce9f9cea89 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:05.935262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a8983b-f3ee-4659-8e10-00ce9f9cea89 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:05.935748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a8983b-f3ee-4659-8e10-00ce9f9cea89 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:05.937322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 50/99] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:05.942875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c197945a-62f0-46d8-9bd9-ab9d5b7c5ab0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:05.943432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c197945a-62f0-46d8-9bd9-ab9d5b7c5ab0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:05.943903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c197945a-62f0-46d8-9bd9-ab9d5b7c5ab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:05.972331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c197945a-62f0-46d8-9bd9-ab9d5b7c5ab0 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:05.974944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c197945a-62f0-46d8-9bd9-ab9d5b7c5ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:06.041884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 50/100] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:06.075030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204483c-e8a8-44b5-862e-bc03eef8115f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.075616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204483c-e8a8-44b5-862e-bc03eef8115f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.076131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204483c-e8a8-44b5-862e-bc03eef8115f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.077698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 51/101] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:06.094868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f8eaaf-794e-46f6-9f6c-8d370a0d103e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.095610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f8eaaf-794e-46f6-9f6c-8d370a0d103e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.096145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f8eaaf-794e-46f6-9f6c-8d370a0d103e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.122926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 51/102] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:06.124246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a8d30-7288-4767-9959-4374806cdb55 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.124536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a8d30-7288-4767-9959-4374806cdb55 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.124816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655a8d30-7288-4767-9959-4374806cdb55 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.126229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 52/103] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:06.126710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f6f8eaaf-794e-46f6-9f6c-8d370a0d103e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:06.128457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f6f8eaaf-794e-46f6-9f6c-8d370a0d103e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:06.137801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b9adab-51e3-4d0b-ac24-1ff55d5f8e3b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.138356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b9adab-51e3-4d0b-ac24-1ff55d5f8e3b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.138820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b9adab-51e3-4d0b-ac24-1ff55d5f8e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.141073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:06.172948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-05b9adab-51e3-4d0b-ac24-1ff55d5f8e3b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:06.175014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-05b9adab-51e3-4d0b-ac24-1ff55d5f8e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:06.266183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 52/104] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:06.300007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 53/105] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:06.528946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f8eaaf-794e-46f6-9f6c-8d370a0d103e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:06.535096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b9adab-51e3-4d0b-ac24-1ff55d5f8e3b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:06.545794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:06.676309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b9adab-51e3-4d0b-ac24-1ff55d5f8e3b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:06.706860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f8eaaf-794e-46f6-9f6c-8d370a0d103e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:06.760019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74cd6195-676b-43ce-9430-2898eaf62eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:06.760668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cd6195-676b-43ce-9430-2898eaf62eb6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.761090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cd6195-676b-43ce-9430-2898eaf62eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.761629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74cd6195-676b-43ce-9430-2898eaf62eb6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.769537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:06.770296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.770581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.770951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.775290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74cd6195-676b-43ce-9430-2898eaf62eb6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-74cd6195-676b-43ce-9430-2898eaf62eb6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:06.780740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:06.781568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:06.782294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 54/106] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:06.783969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663d5af-307f-4374-af04-032b1827dcd5 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.783969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663d5af-307f-4374-af04-032b1827dcd5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.784150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663d5af-307f-4374-af04-032b1827dcd5 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.786574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:06.786725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 55/107] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:06.786725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663d5af-307f-4374-af04-032b1827dcd5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:06.790074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:06.792508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:06.793555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 53/108] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:06.794323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:06.804290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:06.864805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:06.905488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663d5af-307f-4374-af04-032b1827dcd5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:06.910389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b29639-ed51-41bc-ad63-ec55ba80143a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:06.943101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:06.943780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.944099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.944547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.945717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:06.947181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:06.953354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 56/109] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:06.953354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:06.954137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249fd046-ad1b-4ec5-8992-825526c729b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:06.955668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249fd046-ad1b-4ec5-8992-825526c729b8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.956137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249fd046-ad1b-4ec5-8992-825526c729b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.956552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249fd046-ad1b-4ec5-8992-825526c729b8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.968262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249fd046-ad1b-4ec5-8992-825526c729b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-249fd046-ad1b-4ec5-8992-825526c729b8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:06.973511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:06.974397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:06.975149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 54/110] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:06.976260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c505c253-271c-4527-93ac-62d04b4dd208 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:06.976354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c505c253-271c-4527-93ac-62d04b4dd208 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:06.976631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c505c253-271c-4527-93ac-62d04b4dd208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:06.996110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c505c253-271c-4527-93ac-62d04b4dd208 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:06.997296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c505c253-271c-4527-93ac-62d04b4dd208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:07.062602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fc4d87-e68b-44c4-9b4d-a0489f915f00 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.064291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.081362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 55/111] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:07.082508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c505c253-271c-4527-93ac-62d04b4dd208 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.112508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.113162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.113504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.113780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.114453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:07.114764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.116335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.118806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.127520 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 57/112] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:07.128486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.200891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c505c253-271c-4527-93ac-62d04b4dd208 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.234889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.235244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0020c-f72c-400b-a0d7-d9cdacf9e922 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.235521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.236106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.236389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.237054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:07.237353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.238734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.247293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 56/113] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:07.248121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.274907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.275521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.275803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.276109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.277200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.278644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.315078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 58/114] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:40:07 2022] POST /identity/v3/projects => generated 302 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:07.315078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.328045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.368347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.396182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3050453-7968-42e0-8fe3-854b5eb15198 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.427209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7fad52-b29d-4e4d-8e6c-2b4da53e410c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.445139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc598cf-af0d-4254-b270-23219bd56300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.445951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc598cf-af0d-4254-b270-23219bd56300 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.446237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc598cf-af0d-4254-b270-23219bd56300 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.446855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc598cf-af0d-4254-b270-23219bd56300 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.447546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc598cf-af0d-4254-b270-23219bd56300 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:07.447853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc598cf-af0d-4254-b270-23219bd56300 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.449797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc598cf-af0d-4254-b270-23219bd56300 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.470626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 57/115] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:07.470626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5492b4-5117-49d3-a1d7-9998341f54bf None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.470626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5492b4-5117-49d3-a1d7-9998341f54bf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.470953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.471045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5492b4-5117-49d3-a1d7-9998341f54bf None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.471112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.471112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.471112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] PATH_INFO: `/v3/users/a00246e1b29a4746b4e48d4dc1f5618c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.471283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 58/116] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:07.471345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] RBAC: Authorizing `identity:get_user(user_id=a00246e1b29a4746b4e48d4dc1f5618c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.471409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5492b4-5117-49d3-a1d7-9998341f54bf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.471471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.471471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 59/117] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/users/a00246e1b29a4746b4e48d4dc1f5618c => generated 247 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:07.472885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.582694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a155c-9af4-459e-b2d8-a3d385da07b2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.586521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5492b4-5117-49d3-a1d7-9998341f54bf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.601311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.603444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.622099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.622601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0b7b27-aec9-4c54-b3e3-2a64c0f509a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.622785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.623017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.623225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b7b27-aec9-4c54-b3e3-2a64c0f509a5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.623333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.623506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b7b27-aec9-4c54-b3e3-2a64c0f509a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.623764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0b7b27-aec9-4c54-b3e3-2a64c0f509a5 None admin] PATH_INFO: `/v3/users/05266b2196a3433bb9bf46278f536f47` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.635341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:07.637930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0b7b27-aec9-4c54-b3e3-2a64c0f509a5 None admin] RBAC: Authorizing `identity:get_user(user_id=05266b2196a3433bb9bf46278f536f47)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.639385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0b7b27-aec9-4c54-b3e3-2a64c0f509a5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:07.640368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:07.646549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:07.647366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:07.647947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 59/118] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/users/05266b2196a3433bb9bf46278f536f47 => generated 289 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:07.648019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 60/119] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:07.648019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.648148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe8aaa0-8bc2-42fd-a693-43f32db2488d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.648148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe8aaa0-8bc2-42fd-a693-43f32db2488d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.648148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe8aaa0-8bc2-42fd-a693-43f32db2488d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.666238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3fe8aaa0-8bc2-42fd-a693-43f32db2488d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:07.667419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3fe8aaa0-8bc2-42fd-a693-43f32db2488d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:07.738025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 60/120] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:07.738688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe8aaa0-8bc2-42fd-a693-43f32db2488d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.760528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3409b96a-a6a6-4981-a3d6-667affb44dce None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.812768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.814241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.814902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.816126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.828133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.828133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.828133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 61/121] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:07.828133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.857629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe8aaa0-8bc2-42fd-a693-43f32db2488d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.860182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.880914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:07.906621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.908122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.908562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.908970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:07.910248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:07.912163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:07.914545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 61/122] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:07.915517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:07.944047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94f59d-7f65-4241-8c51-397eae091f60 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:07.978985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:07.979603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:07.979886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:07.980191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.028957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2df376-52ea-4170-8427-46d2f51010d8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.056309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.057790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.066218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.066827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.067435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.067559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.068171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:08.068474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.069851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.083524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 62/123] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:08.083524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.107115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:08.120634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:08.172452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 62/124] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:08.172452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.196750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5295b31-16fb-4a40-9be7-fd84c18f694f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.232770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.233423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.233719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.233983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.235199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.236976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.238856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 63/125] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:08 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:08.244021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.274030 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c66595ce-8b02-4406-af74-49a40ccda812 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.308506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.309373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.309841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.310122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/a00246e1b29a4746b4e48d4dc1f5618c/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.312051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] RBAC: Authorizing `identity:create_grant(project_id=5d0bb683d54f429588aceddf17bb9ef8, user_id=a00246e1b29a4746b4e48d4dc1f5618c, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.313467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.327507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 63/126] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:08 2022] PUT /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/a00246e1b29a4746b4e48d4dc1f5618c/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:08.328620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.343977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:08.382143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bd5c18-4505-4f3c-b5f0-2ee64898a3f4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.384408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:08.436501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.437551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.438530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.439374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.456394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:08.463377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be9280-d781-4152-a70f-8c5acfaf19ba None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:08.464562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:08.465408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:08.466225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:08.470330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 64/127] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:08.471492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.514210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.514747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.515071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.515349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.516796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.518773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.520764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 64/128] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:08 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:08.521512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.608758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:08.623243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694659a7-6e0c-44c9-a383-324675b82c9b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.643988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c064f7b9-442a-4fe0-87c9-6950008fa599 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.659656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.660435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.660786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.661212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.662500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.664080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.666241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 65/129] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:08 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:08.667098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.686639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.687554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.687836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.688522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.690880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] RBAC: Authorizing `identity:create_grant(project_id=5d0bb683d54f429588aceddf17bb9ef8, user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.692920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.701797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:08.711081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 65/130] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:08 2022] PUT /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:08.712024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.802266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f77b287-b6e5-4835-9b5d-13380e57f5cf None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.846354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722500a4-17e9-462e-9caf-88b8c284d367 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:08.887522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.887846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:08.888321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.888493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:08.888584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.888748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:08.888840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.889016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:08.889430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:08.889883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.890156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:08.891286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.891597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:08.894655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 66/131] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:08 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:08.895507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:08.898366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:08.918617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 66/132] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:08.919575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:09.032538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cd5287-7ddd-4ac3-ba77-cd301acfacda None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:09.046797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:09.057390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a25f56-6ce7-403d-af76-c025821f9f9c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:09.068807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:09.069720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:09.070217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:09.070680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:09.071730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] RBAC: Adding query filter params (role.id=aca3a5ddd1af426f83c15692fff0a29f, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=05266b2196a3433bb9bf46278f536f47) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:09.072357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:09.073991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:09.080608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 67/133] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:08 2022] GET /identity/v3/role_assignments?user.id=05266b2196a3433bb9bf46278f536f47&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=aca3a5ddd1af426f83c15692fff0a29f => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:09.081812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:09.094282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:09.095038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:09.095433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:09.095901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:09.098352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] RBAC: Authorizing `identity:create_grant(project_id=5d0bb683d54f429588aceddf17bb9ef8, user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797, role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:09.099816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:09.117515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 67/134] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:08 2022] PUT /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797/roles/9d68999390534c65b1cfc53d421f6412 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:09.166822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:09.206153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2569399c-a0e1-4731-898a-f74911d7db48 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:09.246762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:09.247792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:09.248331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:09.248816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:09.250236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:09.252496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:09.254532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 68/135] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:09 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:09.260659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:09.405274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790013f1-11f1-485a-9e26-85e0a184e1a3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:09.444373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959909aa-31d3-4ab2-b305-73291c624832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:09.444943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959909aa-31d3-4ab2-b305-73291c624832 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:09.445225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959909aa-31d3-4ab2-b305-73291c624832 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:09.445499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959909aa-31d3-4ab2-b305-73291c624832 None admin] PATH_INFO: `/v3/projects/94593fc673d64bf6ba670d636a646ce8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:09.446701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959909aa-31d3-4ab2-b305-73291c624832 None admin] RBAC: Authorizing `identity:get_project(project_id=94593fc673d64bf6ba670d636a646ce8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:09.448220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959909aa-31d3-4ab2-b305-73291c624832 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:09.450544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 68/136] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:09 2022] GET /identity/v3/projects/94593fc673d64bf6ba670d636a646ce8 => generated 312 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:09.456884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:09.568441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a08c08d-4b25-4d89-b7eb-a2497bd3e577 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:09.614970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbf3d72-2365-47c3-8613-6cc7aaad1a01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:09.615645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf3d72-2365-47c3-8613-6cc7aaad1a01 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:09.615926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf3d72-2365-47c3-8613-6cc7aaad1a01 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:09.616271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf3d72-2365-47c3-8613-6cc7aaad1a01 None admin] PATH_INFO: `/v3/projects/94593fc673d64bf6ba670d636a646ce8/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:09.618193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbf3d72-2365-47c3-8613-6cc7aaad1a01 None admin] RBAC: Authorizing `identity:create_grant(project_id=94593fc673d64bf6ba670d636a646ce8, user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:09.619628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbf3d72-2365-47c3-8613-6cc7aaad1a01 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:09.622066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:09.643715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 69/137] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:09 2022] PUT /identity/v3/projects/94593fc673d64bf6ba670d636a646ce8/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:09.853190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff234a4-cf99-44b0-8759-ed950ec21063 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:09.853523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff234a4-cf99-44b0-8759-ed950ec21063 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:09.853754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff234a4-cf99-44b0-8759-ed950ec21063 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:09.855098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 69/138] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:09.860663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a78b537-ac9c-4017-90ad-f07418fde51d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:09.860963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a78b537-ac9c-4017-90ad-f07418fde51d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:09.861221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a78b537-ac9c-4017-90ad-f07418fde51d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:09.884105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-2a78b537-ac9c-4017-90ad-f07418fde51d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:09.885369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2a78b537-ac9c-4017-90ad-f07418fde51d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:09.976546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 70/139] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:10.336444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff234a4-cf99-44b0-8759-ed950ec21063 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:10.360041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:10.467045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff234a4-cf99-44b0-8759-ed950ec21063 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:10.508410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826cde00-acc9-4398-bcfb-91075125d289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:10.509376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826cde00-acc9-4398-bcfb-91075125d289 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:10.509846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826cde00-acc9-4398-bcfb-91075125d289 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:10.510348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826cde00-acc9-4398-bcfb-91075125d289 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:10.512789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826cde00-acc9-4398-bcfb-91075125d289 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:10.514404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826cde00-acc9-4398-bcfb-91075125d289 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:10.516470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 70/140] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:10.523004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a78b537-ac9c-4017-90ad-f07418fde51d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:10.644445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a78b537-ac9c-4017-90ad-f07418fde51d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:10.679550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:10.691892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65428639-ccc7-41c4-a92d-f19e27a8d990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:10.692810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65428639-ccc7-41c4-a92d-f19e27a8d990 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:10.693279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65428639-ccc7-41c4-a92d-f19e27a8d990 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:10.693738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65428639-ccc7-41c4-a92d-f19e27a8d990 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:10.694607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65428639-ccc7-41c4-a92d-f19e27a8d990 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:10.696262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65428639-ccc7-41c4-a92d-f19e27a8d990 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:10.741298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 71/141] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:10 2022] POST /identity/v3/users => generated 283 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:11.584992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fbf4ab-b7f7-4794-90bf-e72d96560e11 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:11.585979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fbf4ab-b7f7-4794-90bf-e72d96560e11 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:11.586537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fbf4ab-b7f7-4794-90bf-e72d96560e11 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:11.588097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 71/142] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:11.593570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68dbaed-b3c5-4386-a0f5-c813cfd40a32 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:11.593978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68dbaed-b3c5-4386-a0f5-c813cfd40a32 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:11.594160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68dbaed-b3c5-4386-a0f5-c813cfd40a32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:11.616399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b68dbaed-b3c5-4386-a0f5-c813cfd40a32 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:11.617585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b68dbaed-b3c5-4386-a0f5-c813cfd40a32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:11.702220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 72/143] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:11.800410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079def2-5140-4093-9229-a581ad3d3a62 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:11.800734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079def2-5140-4093-9229-a581ad3d3a62 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:11.800961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3079def2-5140-4093-9229-a581ad3d3a62 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:11.802464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 72/144] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:11.808206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76da5e8-aec8-4b59-9e74-bc76d4661398 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:11.808497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76da5e8-aec8-4b59-9e74-bc76d4661398 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:11.808772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76da5e8-aec8-4b59-9e74-bc76d4661398 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:11.833368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e76da5e8-aec8-4b59-9e74-bc76d4661398 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:11.834555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e76da5e8-aec8-4b59-9e74-bc76d4661398 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:11.915466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 73/145] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:12.128203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46330db7-bffa-4018-97c1-ac4f26ef45a4 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:12.128558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46330db7-bffa-4018-97c1-ac4f26ef45a4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:12.128766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46330db7-bffa-4018-97c1-ac4f26ef45a4 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:12.130455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 73/146] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:12.135644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9b859d-b50b-41ab-be41-35fc9bf86c80 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:12.135989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9b859d-b50b-41ab-be41-35fc9bf86c80 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:12.136239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9b859d-b50b-41ab-be41-35fc9bf86c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:12.155472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8f9b859d-b50b-41ab-be41-35fc9bf86c80 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:12.156671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8f9b859d-b50b-41ab-be41-35fc9bf86c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:12.209508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:12.221309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46330db7-bffa-4018-97c1-ac4f26ef45a4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:12.251407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 74/147] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:12.281711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9b859d-b50b-41ab-be41-35fc9bf86c80 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:12.387696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:12.422449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46330db7-bffa-4018-97c1-ac4f26ef45a4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:12.481420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9b859d-b50b-41ab-be41-35fc9bf86c80 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:12.499181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:12.500191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:12.500659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:12.501652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:12.527996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:12.534066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:12.534904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:12.535638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 74/148] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:12.543969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:12.584760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:12.585714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:12.586187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:12.586622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:12.613016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:12.618698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:12.619543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:12.620321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:12.622886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 75/149] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 342 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:12.630260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:12.700397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:12.727239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db49ee03-1d43-4845-a25c-a1fcafd76d28 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:12.763067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e5399d-dbc4-42fc-b160-c02623c1688e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:12.793210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:12.793811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:12.794087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:12.794350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:12.795015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:12.795308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:12.802625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:12.804657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:12.805449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:12.805987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:12.806398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:12.807208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:12.807739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:12.809941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:12.819077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 76/150] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:12.824189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 75/151] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:12.825857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:12.831895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:12.859320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:12.990227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:12.998445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3389c87-428e-43b8-b22b-80219f783e85 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:13.037265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:13.038264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b81f59-cb58-44b2-8fd2-da7dadf53220 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:13.038901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.039364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] PATH_INFO: `/v3/users/a00246e1b29a4746b4e48d4dc1f5618c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.049897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] RBAC: Authorizing `identity:get_user(user_id=a00246e1b29a4746b4e48d4dc1f5618c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:13.051698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:13.054183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 77/152] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:12 2022] GET /identity/v3/users/a00246e1b29a4746b4e48d4dc1f5618c => generated 247 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:13.055164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:13.153560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5cfd4d-37f3-4faf-8f97-736ac080450d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:13.154212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cfd4d-37f3-4faf-8f97-736ac080450d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.154493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cfd4d-37f3-4faf-8f97-736ac080450d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.154761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cfd4d-37f3-4faf-8f97-736ac080450d None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.155952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5cfd4d-37f3-4faf-8f97-736ac080450d None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:13.158030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5cfd4d-37f3-4faf-8f97-736ac080450d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:13.164056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 76/153] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:12 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:13.165433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d31fb9-38e3-4213-a794-349ebb12788d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.165804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d31fb9-38e3-4213-a794-349ebb12788d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.166254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d31fb9-38e3-4213-a794-349ebb12788d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.167380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 77/154] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:13.172213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d31fb9-38e3-4213-a794-349ebb12788d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:13.314661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222a5a9d-3834-4861-a17b-a2940bbed574 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:13.323516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:13.372848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d31fb9-38e3-4213-a794-349ebb12788d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:13.388089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:13.388925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.389203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.389466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.420427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:13.431111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:13.432039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:13.432960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:13.440767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 78/155] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:13 2022] GET /identity/v3/roles/reader => generated 84 bytes in 386 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:13.441754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:13.448743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4403d18-bd38-45fa-a420-15f6570385e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:13.449680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4403d18-bd38-45fa-a420-15f6570385e7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.450161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4403d18-bd38-45fa-a420-15f6570385e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.450629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4403d18-bd38-45fa-a420-15f6570385e7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.472727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:13.582309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4403d18-bd38-45fa-a420-15f6570385e7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:13.583708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4403d18-bd38-45fa-a420-15f6570385e7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:13.617635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae77b8f-2930-45bb-9f17-73cb8e2fd9ca None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:13.680372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c970a6-50bd-498b-8363-8c6ee1091dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:13.680985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c970a6-50bd-498b-8363-8c6ee1091dfd None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.681260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c970a6-50bd-498b-8363-8c6ee1091dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.681526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c970a6-50bd-498b-8363-8c6ee1091dfd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.732307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:13.774372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 78/156] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 606 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:13.776675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dc316-19fe-47c0-b314-7999a02108b5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.776954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dc316-19fe-47c0-b314-7999a02108b5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.777216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909dc316-19fe-47c0-b314-7999a02108b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.803971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-909dc316-19fe-47c0-b314-7999a02108b5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:13.803971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-909dc316-19fe-47c0-b314-7999a02108b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:13.819854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c970a6-50bd-498b-8363-8c6ee1091dfd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:13.821434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c970a6-50bd-498b-8363-8c6ee1091dfd None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:13.862331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:13.924675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 79/157] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:13.925613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909dc316-19fe-47c0-b314-7999a02108b5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:13.990933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 79/158] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 550 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:13.992762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe47de2-b30a-48e6-ae81-a2aed442239a None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:13.993258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe47de2-b30a-48e6-ae81-a2aed442239a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:13.993915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe47de2-b30a-48e6-ae81-a2aed442239a None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:13.995424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 80/159] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:13.996364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe47de2-b30a-48e6-ae81-a2aed442239a None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:14.164733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909dc316-19fe-47c0-b314-7999a02108b5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:14.166468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe47de2-b30a-48e6-ae81-a2aed442239a None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:14.181534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:14.199989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:14.226091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e58651-5f05-4b3c-ae1f-4e44305f5aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:14.226690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e58651-5f05-4b3c-ae1f-4e44305f5aef None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:14.226967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e58651-5f05-4b3c-ae1f-4e44305f5aef None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:14.227231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e58651-5f05-4b3c-ae1f-4e44305f5aef None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:14.227886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e58651-5f05-4b3c-ae1f-4e44305f5aef None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:14.228202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e58651-5f05-4b3c-ae1f-4e44305f5aef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:14.229570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e58651-5f05-4b3c-ae1f-4e44305f5aef None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:14.233621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:14.234246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:14.234529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:14.234791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:14.237229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:14.238633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:14.243124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 81/160] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:13 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:14.244672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:14.249370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 80/161] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:13 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:14.251658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753a5738-e170-480b-bc28-1b1b44b554cc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:14.251929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753a5738-e170-480b-bc28-1b1b44b554cc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:14.252213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753a5738-e170-480b-bc28-1b1b44b554cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:14.278322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-753a5738-e170-480b-bc28-1b1b44b554cc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:14.279485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-753a5738-e170-480b-bc28-1b1b44b554cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:14.410736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 81/162] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:14.417472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753a5738-e170-480b-bc28-1b1b44b554cc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:14.434135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556c3977-37a5-4ac4-bf57-547c885ec4ee None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:14.541555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:14.542200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:14.542484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:14.542754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:14.548734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:14.550212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:14.551926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 82/163] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:14 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:14.552719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:14.569400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:14.573454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753a5738-e170-480b-bc28-1b1b44b554cc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:14.637966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:14.638570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:14.638845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:14.639108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:14.639830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=a00246e1b29a4746b4e48d4dc1f5618c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:14.640155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:14.642192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:14.655441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:14.656191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 82/164] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:14 2022] GET /identity/v3/role_assignments?user.id=a00246e1b29a4746b4e48d4dc1f5618c&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:14.657350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:14.743333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca25aa6-06d9-469b-88d8-2922f028aa2e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:14.853265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:14.853916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:14.854189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:14.854458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] PATH_INFO: `/v3/users/05266b2196a3433bb9bf46278f536f47` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:14.860876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] RBAC: Authorizing `identity:get_user(user_id=05266b2196a3433bb9bf46278f536f47)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:14.862417 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:14.864164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 83/165] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:14 2022] GET /identity/v3/users/05266b2196a3433bb9bf46278f536f47 => generated 289 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:14.864973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:14.866301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6dc0b-61a9-4940-9afc-76c043c2ef8c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:14.949174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:14.949816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:14.950085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:14.950345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:14.976921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:14.982274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:14.983150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:14.984229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:14.987286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 83/166] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:14.988137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.027119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db183af-5152-4257-b4ed-a25d8035a6cc None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.045973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:15.059493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:15.084792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.085428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.085708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.085977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.086733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:15.087021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:15.088513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:15.098190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 84/167] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:14 2022] GET /identity/v3/role_assignments?user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 608 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:15.099127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.177470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6fe77f-05c8-43eb-8fa5-bf01194164a4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.257448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.258175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.258462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.258732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.275582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:15.289391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:15.290308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:15.291152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 84/168] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:15.291814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.317151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876cf00a-faf4-4cf0-8091-8d526c5b8a6d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.355091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d18b47-e38c-4700-b845-adf999469b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.355744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d18b47-e38c-4700-b845-adf999469b27 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.356043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d18b47-e38c-4700-b845-adf999469b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.356312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d18b47-e38c-4700-b845-adf999469b27 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.357483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d18b47-e38c-4700-b845-adf999469b27 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:15.358927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d18b47-e38c-4700-b845-adf999469b27 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:15.360803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 85/169] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:15 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:15.362312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed7244f-05f0-4bcc-8b35-51ee7128ceb4 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.362590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed7244f-05f0-4bcc-8b35-51ee7128ceb4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.362857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed7244f-05f0-4bcc-8b35-51ee7128ceb4 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.364108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 86/170] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:15.367476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed7244f-05f0-4bcc-8b35-51ee7128ceb4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.400422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:15.418940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4b1adc-614c-45d2-8f8d-64e1d195d903 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.431897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:15.472779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768816f7-a9a3-418a-be9b-532e84b248e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.473449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768816f7-a9a3-418a-be9b-532e84b248e1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.473731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768816f7-a9a3-418a-be9b-532e84b248e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.474001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768816f7-a9a3-418a-be9b-532e84b248e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.474670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768816f7-a9a3-418a-be9b-532e84b248e1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:15.474967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768816f7-a9a3-418a-be9b-532e84b248e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:15.476413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768816f7-a9a3-418a-be9b-532e84b248e1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:15.484448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed7244f-05f0-4bcc-8b35-51ee7128ceb4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.487604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 85/171] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:15.488983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a469011-3e8f-46bc-822b-5db79ac2b7b1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.489268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a469011-3e8f-46bc-822b-5db79ac2b7b1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.489538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a469011-3e8f-46bc-822b-5db79ac2b7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.515112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3a469011-3e8f-46bc-822b-5db79ac2b7b1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:15.519432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3a469011-3e8f-46bc-822b-5db79ac2b7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:15.525519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.526133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.526410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.526962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.527448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:15.527749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:15.529247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:15.541765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 87/172] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:15.542505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.588426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 86/173] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:15.596447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bb1353-ef03-46f8-ae0e-aa37703087c9 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.596447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bb1353-ef03-46f8-ae0e-aa37703087c9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.596447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bb1353-ef03-46f8-ae0e-aa37703087c9 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.596447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 87/174] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:15.596447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bb1353-ef03-46f8-ae0e-aa37703087c9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.688133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:15.694469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bcee1c-d1be-4932-9641-97d7788cdbd7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.714931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:15.724953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bb1353-ef03-46f8-ae0e-aa37703087c9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.738119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.738990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.739183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.739429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/05266b2196a3433bb9bf46278f536f47/roles/aca3a5ddd1af426f83c15692fff0a29f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.741444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] RBAC: Authorizing `identity:create_grant(project_id=5d0bb683d54f429588aceddf17bb9ef8, user_id=05266b2196a3433bb9bf46278f536f47, role_id=aca3a5ddd1af426f83c15692fff0a29f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:15.742969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:15.758764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 88/175] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:15 2022] PUT /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/users/05266b2196a3433bb9bf46278f536f47/roles/aca3a5ddd1af426f83c15692fff0a29f => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:15.764882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.771494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92649e8c-634d-465e-97a6-90bdf949c3a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.772581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92649e8c-634d-465e-97a6-90bdf949c3a9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.773093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92649e8c-634d-465e-97a6-90bdf949c3a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.773577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92649e8c-634d-465e-97a6-90bdf949c3a9 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.775046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92649e8c-634d-465e-97a6-90bdf949c3a9 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:15.776879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92649e8c-634d-465e-97a6-90bdf949c3a9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:15.778816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 88/176] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:15 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:15.785550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb1f091-0747-4914-bd55-3248ebe4ee0c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.786039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb1f091-0747-4914-bd55-3248ebe4ee0c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.786513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb1f091-0747-4914-bd55-3248ebe4ee0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.818052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6cb1f091-0747-4914-bd55-3248ebe4ee0c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:15.819619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6cb1f091-0747-4914-bd55-3248ebe4ee0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:15.896249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982be997-449d-42d6-90ac-e88c8c2474a4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:15.896936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 89/177] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:15.898480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f259e62-cf51-47a8-b847-f2217112b1eb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.898763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f259e62-cf51-47a8-b847-f2217112b1eb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.899045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f259e62-cf51-47a8-b847-f2217112b1eb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.903994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 90/178] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:15.903994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f259e62-cf51-47a8-b847-f2217112b1eb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:15.949271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:15.949904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:15.950184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:15.950450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] PATH_INFO: `/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:15.952526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] RBAC: Authorizing `identity:get_user(user_id=a5d7b215b8b94e8d8b6ec1b2b1dc9797)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:15.953095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:15.955107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 89/179] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:15 2022] GET /identity/v3/users/a5d7b215b8b94e8d8b6ec1b2b1dc9797 => generated 275 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:15.955713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.003782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.017273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.037880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f259e62-cf51-47a8-b847-f2217112b1eb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.074918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2055c128-4169-4380-8118-534a7d540540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.075436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2055c128-4169-4380-8118-534a7d540540 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.075708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2055c128-4169-4380-8118-534a7d540540 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.076016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2055c128-4169-4380-8118-534a7d540540 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.084468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc6b323-fd89-46c2-a400-3f3b7a6d2661 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.130006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038b4f88-c36f-42fd-aaae-2daac8775f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.130890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b4f88-c36f-42fd-aaae-2daac8775f97 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.131169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b4f88-c36f-42fd-aaae-2daac8775f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.131819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038b4f88-c36f-42fd-aaae-2daac8775f97 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.145391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038b4f88-c36f-42fd-aaae-2daac8775f97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:16.150833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2055c128-4169-4380-8118-534a7d540540 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-038b4f88-c36f-42fd-aaae-2daac8775f97 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:16.151541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:16.152399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:16.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:16.153735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2055c128-4169-4380-8118-534a7d540540 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:16.153806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 90/180] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:16.154352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d16b8c-6b58-4b10-8758-60670de2b924 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.154499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d16b8c-6b58-4b10-8758-60670de2b924 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.157519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d16b8c-6b58-4b10-8758-60670de2b924 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.175820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d5d16b8c-6b58-4b10-8758-60670de2b924 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:16.177303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d5d16b8c-6b58-4b10-8758-60670de2b924 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:16.234863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.258529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 91/181] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:16.259271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d16b8c-6b58-4b10-8758-60670de2b924 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.263975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 91/182] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:16.263975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2055c128-4169-4380-8118-534a7d540540 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.303841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.372678 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2055c128-4169-4380-8118-534a7d540540 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.382929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d16b8c-6b58-4b10-8758-60670de2b924 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.421249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.422187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.423232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.423334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.423801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.424048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.424301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.424573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.425332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:16.425638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:16.427007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:16.436981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 92/183] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:16.442898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.479459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.510027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:16.512115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:16.548895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8e93d2-5030-4597-8ded-72f8b7d17d11 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.553641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.585372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.586050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.586332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.586594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.597017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:16.602273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:16.603137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:16.603927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 93/184] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:16.605069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.628601 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 92/185] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:16.629707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.711559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.718156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5374990-a6bb-40ea-b163-881309dcb88b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.739891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55f9a87-9bf9-44b9-9ab9-f69cd352f8ce None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.755118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.755747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.756050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.756319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.757466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:16.758911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:16.765711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 94/186] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:16.765711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.780839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.781840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.782736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.783147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] PATH_INFO: `/v3/users/d06cea73fe1845a985688560b2baa9ee` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.786759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.803830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] RBAC: Authorizing `identity:get_user(user_id=d06cea73fe1845a985688560b2baa9ee)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:16.805818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:16.807570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 93/187] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/users/d06cea73fe1845a985688560b2baa9ee => generated 283 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:16.808460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.881351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca9520-f5f0-439f-9630-39fb176ebfc7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.919148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.919759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.920051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.920306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.921385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:16.922058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec5c9a1-d416-429d-83d6-67113682a57c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:16.922900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:16.925298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 95/188] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:16.925409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:16.959667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:16.965532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:16.966145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:16.966426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:16.966697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:16.967367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:16.967668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:16.969262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:16.977632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 94/189] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:16.982807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.033621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76042772-f953-42fe-a0c7-acbbe2f2fc8d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.042098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:17.070188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.070808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.071085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.071356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] PATH_INFO: `/v3/projects/94593fc673d64bf6ba670d636a646ce8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.072498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] RBAC: Authorizing `identity:get_project(project_id=94593fc673d64bf6ba670d636a646ce8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.073949 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.075565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 96/190] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/projects/94593fc673d64bf6ba670d636a646ce8 => generated 312 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:17.076360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.097121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156b33e7-e577-44c3-95bc-fba33ceec497 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.132895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.133593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.133871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.134136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.134853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] RBAC: Adding query filter params (role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:17.135157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.136607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.143198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 95/191] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:16 2022] GET /identity/v3/role_assignments?user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=9d68999390534c65b1cfc53d421f6412 => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:17.148334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.213816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:17.219605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148d3783-d422-445a-8fc3-ca55bce745fc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.269818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8aa026-16f5-4524-9cc9-f07799bae401 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.271977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.273976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.274514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.274984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.298918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:17.323879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.324523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.324805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.325727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.325727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.334047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.354923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 96/192] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:17 2022] POST /identity/v3/users => generated 297 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:17.356228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.363049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.365056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.504276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:17.510433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 97/193] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:17.511571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.527654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d48d3c5-ede2-4725-bc2e-9a22afc0f69f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.576835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.577476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.577761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.578039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] PATH_INFO: `/v3/users/d06cea73fe1845a985688560b2baa9ee` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.579268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] RBAC: Authorizing `identity:get_user(user_id=d06cea73fe1845a985688560b2baa9ee)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.580737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.583524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 97/194] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:17 2022] GET /identity/v3/users/d06cea73fe1845a985688560b2baa9ee => generated 283 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:17.584444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.632985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cfa4fa-58a7-4250-9d68-3f97005f0d68 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.640052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:17.680892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.681511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.682029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.682029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.683914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=94593fc673d64bf6ba670d636a646ce8, user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:17.683914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.684533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.692919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 98/195] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:17 2022] GET /identity/v3/role_assignments?user.id=a5d7b215b8b94e8d8b6ec1b2b1dc9797&scope.project.id=94593fc673d64bf6ba670d636a646ce8&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:17.693154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.716953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e793536-6778-4217-9ceb-c44775e27a82 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.766347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.766980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.767195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.767525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.769103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.771382 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.773578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 98/196] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:17 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:17.780903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:17.781469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:17.833507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7a0f52-9c02-4271-a9b7-413824fe3620 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.888488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc3b9d0-1d12-4864-abd9-b4354a8b65cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.889348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3b9d0-1d12-4864-abd9-b4354a8b65cd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.889775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3b9d0-1d12-4864-abd9-b4354a8b65cd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.890187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3b9d0-1d12-4864-abd9-b4354a8b65cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.925596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef7d0a-5ac7-46af-818b-ca0794bbf943 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:17.943139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:17.975093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:17.976007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:17.976443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:17.976872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:17.977900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=d06cea73fe1845a985688560b2baa9ee) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:17.978355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.980128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.983189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc3b9d0-1d12-4864-abd9-b4354a8b65cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:17.984753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc3b9d0-1d12-4864-abd9-b4354a8b65cd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:17.987657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 99/197] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:17 2022] GET /identity/v3/role_assignments?user.id=d06cea73fe1845a985688560b2baa9ee&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:18.068450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:18.110879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 99/198] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:18.115411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:18.142384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e1da9-853a-496b-9c25-29b21185a10b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:18.143239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e1da9-853a-496b-9c25-29b21185a10b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:18.143751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e1da9-853a-496b-9c25-29b21185a10b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:18.146626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 100/199] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:18 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:18.154245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3f508-d2b4-44f8-9f8d-b7ed3a25989e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:18.154869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3f508-d2b4-44f8-9f8d-b7ed3a25989e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:18.155384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a3f508-d2b4-44f8-9f8d-b7ed3a25989e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:18.185042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f1a3f508-d2b4-44f8-9f8d-b7ed3a25989e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:18.186606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f1a3f508-d2b4-44f8-9f8d-b7ed3a25989e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:18.275730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd382e9e-d200-408a-a6aa-21fd74e80085 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:18.279666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 101/200] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:18.320683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:18.321243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:18.321518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:18.321787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:18.327402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:18.329596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:18.331637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 100/201] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:18 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:18.337321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a3f508-d2b4-44f8-9f8d-b7ed3a25989e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:18.491575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:18.495104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a3f508-d2b4-44f8-9f8d-b7ed3a25989e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:18.541642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:18.542250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:18.542525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:18.542794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:18.543520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] RBAC: Adding query filter params (role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=d06cea73fe1845a985688560b2baa9ee) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:18.543829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:18.545323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:18.556202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 102/202] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:18 2022] GET /identity/v3/role_assignments?user.id=d06cea73fe1845a985688560b2baa9ee&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=9d68999390534c65b1cfc53d421f6412 => generated 259 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:18.619266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:18.686896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:18.772404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66b3be6-c629-4454-8e65-1029b86a3c3d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:18.818160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:18.818819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:18.819153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:18.819423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:18.834362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:18.840169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:18.841002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:18.841776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:18.844100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 101/203] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:18.853024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:19.014185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209f091-b8f5-446e-9037-0b13b1cd6766 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:19.058571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:19.059190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:19.059468 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:19.059737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:19.060488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:19.060793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:19.062166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:19.074846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 103/204] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:19.080810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:19.233751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b8f3e1-92f4-4779-b08f-c71f93a68968 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:19.240972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:19.285915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:19.286470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:19.286750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:19.287022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] PATH_INFO: `/v3/users/05266b2196a3433bb9bf46278f536f47` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:19.288196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] RBAC: Authorizing `identity:get_user(user_id=05266b2196a3433bb9bf46278f536f47)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:19.289624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:19.291362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 102/205] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:19 2022] GET /identity/v3/users/05266b2196a3433bb9bf46278f536f47 => generated 289 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:19.297105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:19.335853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:19.444078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b268b1c-8167-40f8-8264-2aa18810d08a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:19.496656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:19.497273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:19.497553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:19.497820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:19.583138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:19.584939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:19.619965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:19.719230 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 104/206] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:19.725124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:19.867966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c8ca44-a70a-446c-83bb-2aa3689c3d63 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:19.912563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daec59a-2e36-48dd-a74c-8499a4abb214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:19.913194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daec59a-2e36-48dd-a74c-8499a4abb214 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:19.913470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daec59a-2e36-48dd-a74c-8499a4abb214 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:19.913733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daec59a-2e36-48dd-a74c-8499a4abb214 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:19.914834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daec59a-2e36-48dd-a74c-8499a4abb214 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:19.916280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daec59a-2e36-48dd-a74c-8499a4abb214 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:19.917895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 103/207] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:19 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:19.923815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:20.075703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b255e8db-1d06-47b9-8024-282e44b8f80f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:20.102392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:20.117333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f78da7-a568-47dd-aa53-38daa5eade36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:20.118321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f78da7-a568-47dd-aa53-38daa5eade36 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:20.118801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f78da7-a568-47dd-aa53-38daa5eade36 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:20.119264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f78da7-a568-47dd-aa53-38daa5eade36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:20.120246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f78da7-a568-47dd-aa53-38daa5eade36 None admin] RBAC: Adding query filter params (role.id=aca3a5ddd1af426f83c15692fff0a29f, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8, user.id=05266b2196a3433bb9bf46278f536f47) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:20.120750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f78da7-a568-47dd-aa53-38daa5eade36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:20.122381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f78da7-a568-47dd-aa53-38daa5eade36 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:20.129241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 105/208] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:19 2022] GET /identity/v3/role_assignments?user.id=05266b2196a3433bb9bf46278f536f47&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=aca3a5ddd1af426f83c15692fff0a29f => generated 608 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:20.582769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be46084c-a324-464b-9207-6fced4815f83 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:20.582950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be46084c-a324-464b-9207-6fced4815f83 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:20.583227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be46084c-a324-464b-9207-6fced4815f83 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:20.584774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 104/209] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:20.590887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85383188-6773-40b8-8cce-61836dea5a40 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:20.591173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85383188-6773-40b8-8cce-61836dea5a40 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:20.591441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85383188-6773-40b8-8cce-61836dea5a40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:20.613748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-85383188-6773-40b8-8cce-61836dea5a40 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:20.615273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-85383188-6773-40b8-8cce-61836dea5a40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:20.633221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a91636-45eb-446a-a277-5f6391f4c89a None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:20.633870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a91636-45eb-446a-a277-5f6391f4c89a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:20.634393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a91636-45eb-446a-a277-5f6391f4c89a None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:20.637312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 105/210] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:20.644288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ac869-051d-405e-9199-2b7098296ef6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:20.644851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ac869-051d-405e-9199-2b7098296ef6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:20.645295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ac869-051d-405e-9199-2b7098296ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:20.667818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-ad9ac869-051d-405e-9199-2b7098296ef6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:20.670337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ad9ac869-051d-405e-9199-2b7098296ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:20.699467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 106/211] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:20.721149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:20.768722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 106/212] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:20.848079 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f54fbc4-48e8-4a15-b80a-0a1ee58cd2cf None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:20.848702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f54fbc4-48e8-4a15-b80a-0a1ee58cd2cf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:20.849173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f54fbc4-48e8-4a15-b80a-0a1ee58cd2cf None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:20.850719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 107/213] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:20.856208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6bcdd0-ca64-400d-bd7d-05df7210341d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:20.856747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6bcdd0-ca64-400d-bd7d-05df7210341d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:20.857239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6bcdd0-ca64-400d-bd7d-05df7210341d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:20.877344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6c6bcdd0-ca64-400d-bd7d-05df7210341d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:20.879245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6c6bcdd0-ca64-400d-bd7d-05df7210341d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:20.947900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 107/214] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:21.020045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f54fbc4-48e8-4a15-b80a-0a1ee58cd2cf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.058112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6bcdd0-ca64-400d-bd7d-05df7210341d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.129748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f54fbc4-48e8-4a15-b80a-0a1ee58cd2cf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.147232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:21.169013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:21.170708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a818e4be-316d-4170-8f73-06b46c1c4d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.171336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a818e4be-316d-4170-8f73-06b46c1c4d52 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.171609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a818e4be-316d-4170-8f73-06b46c1c4d52 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.171874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a818e4be-316d-4170-8f73-06b46c1c4d52 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.172073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6bcdd0-ca64-400d-bd7d-05df7210341d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.182332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a818e4be-316d-4170-8f73-06b46c1c4d52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a818e4be-316d-4170-8f73-06b46c1c4d52 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:21.187216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:21.188164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 108/215] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:21.192894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6752e10e-c2ed-4654-b719-5a566bc21872 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.193583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6752e10e-c2ed-4654-b719-5a566bc21872 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.193583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6752e10e-c2ed-4654-b719-5a566bc21872 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.193583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 109/216] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:21.196718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6752e10e-c2ed-4654-b719-5a566bc21872 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.217299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784379c3-688f-416e-879c-d8a961991685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.217923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784379c3-688f-416e-879c-d8a961991685 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.218240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784379c3-688f-416e-879c-d8a961991685 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.218501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784379c3-688f-416e-879c-d8a961991685 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.220473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784379c3-688f-416e-879c-d8a961991685 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:21.221905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784379c3-688f-416e-879c-d8a961991685 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:21.223513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 108/217] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:21.224650 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805512f2-209f-4aae-b1fb-619b93d94691 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.224930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805512f2-209f-4aae-b1fb-619b93d94691 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.225257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805512f2-209f-4aae-b1fb-619b93d94691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.243607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-805512f2-209f-4aae-b1fb-619b93d94691 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:21.244858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-805512f2-209f-4aae-b1fb-619b93d94691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:21.303074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6752e10e-c2ed-4654-b719-5a566bc21872 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.312182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 109/218] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:21.313022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805512f2-209f-4aae-b1fb-619b93d94691 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.337212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.337817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.338093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.338361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.339025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:21.339323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:21.340727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:21.353951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 110/219] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:21.353951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.389899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:21.429738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:21.433775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805512f2-209f-4aae-b1fb-619b93d94691 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.458828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2c4d9e-0085-4ac2-be06-69ccb999daef None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.470997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.471602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.471881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.472183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.472823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:21.474242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:21.495400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.496067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.496339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.496608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.504341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 110/220] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:21 2022] POST /identity/v3/users => generated 297 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:21.505739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.509999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:21.516618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:21.517601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:21.518409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:21.525724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 111/221] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:21.526944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.638090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:21.640831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf883474-c418-4692-a7d7-56224d7bdf81 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.645775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7f4a5a-0732-48c9-957f-a4d483166072 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.677524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.678162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.678430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.678697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] PATH_INFO: `/v3/users/d50463a9b78b40338679be4183c85c5c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.687064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.688236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.688236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.688747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.689641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:21.691326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:21.691512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:21.695865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] RBAC: Authorizing `identity:get_user(user_id=d50463a9b78b40338679be4183c85c5c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:21.697426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:21.699069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 111/222] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/users/d50463a9b78b40338679be4183c85c5c => generated 297 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:21.699833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.702444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 112/223] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:21.706798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.717885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:21.828805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a568ae-edeb-46ae-8138-25aad2928c3d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.834571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f663370-66ab-406e-9bb2-665ef087731e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:21.875814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.876209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.876640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.876911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.877666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:21.878280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:21.878546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:21.879008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:21.892013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:21.897671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:21.898539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:21.899314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:21.901364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 112/224] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:21.902378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:21.916805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:21.969324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:21.971677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:21.981238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:22.013645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa64b607-596d-4e82-9bb6-f128ab62f4dd None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.051246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.052247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.052726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.053217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] PATH_INFO: `/v3/users/d06cea73fe1845a985688560b2baa9ee` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.054542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] RBAC: Authorizing `identity:get_user(user_id=d06cea73fe1845a985688560b2baa9ee)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.056222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.058064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 113/225] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/users/d06cea73fe1845a985688560b2baa9ee => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:22.059028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.108759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 113/226] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:40:22.108759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.173635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12ee94-91f1-4b76-8b6d-bd9cafaec8b9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.176060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:22.211461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.212462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.212966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.213433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.214303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:22.214799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.216424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.220966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093afcc8-e99b-461b-aa9e-5bad72827af5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.226097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 114/227] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:22.227194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.230000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:22.262527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.263127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.263397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.263658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.264817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.266438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.268547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 114/228] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:22 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:22.269323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.338172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b7e3f-31d5-4e7c-ad35-10a14a4aeceb None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.375034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.375234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed238fa4-6ff6-4365-9334-aab20c56d36e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.376758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.377465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.377992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.379386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.381117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.383312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 115/229] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:22 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:22.384353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.408639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:22.420456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.421091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.421362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.421623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] PATH_INFO: `/v3/users/d06cea73fe1845a985688560b2baa9ee` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.423343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] RBAC: Authorizing `identity:get_user(user_id=d06cea73fe1845a985688560b2baa9ee)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.424864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.426717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 115/230] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:22 2022] GET /identity/v3/users/d06cea73fe1845a985688560b2baa9ee => generated 283 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:22.431948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.465497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:22.501159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4b2b53-30ec-4568-a1bf-2356c1178961 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.538863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.539975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.540467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.541042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/d06cea73fe1845a985688560b2baa9ee/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.543512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] RBAC: Authorizing `identity:create_grant(project_id=434cda3cc623466c9163ef469e5b2e7d, user_id=d06cea73fe1845a985688560b2baa9ee, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.544456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5dc825-787e-43ea-82b8-2335d4060bbe None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.545558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.563565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 116/231] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:22 2022] PUT /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/d06cea73fe1845a985688560b2baa9ee/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:22.564990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.589560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.590161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.590440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.590692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.591411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=d50463a9b78b40338679be4183c85c5c) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:22.591708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.593322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.600329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 116/232] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:22 2022] GET /identity/v3/role_assignments?user.id=d50463a9b78b40338679be4183c85c5c&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 259 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:22.705624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6d7432-f58f-4ef2-b78d-c2a78430fa50 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.745067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:22.753232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0bc78c-2f7a-48bf-abe6-6a561d26ebb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.754729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0bc78c-2f7a-48bf-abe6-6a561d26ebb7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.755230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0bc78c-2f7a-48bf-abe6-6a561d26ebb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.755811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0bc78c-2f7a-48bf-abe6-6a561d26ebb7 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.757599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0bc78c-2f7a-48bf-abe6-6a561d26ebb7 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.759422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0bc78c-2f7a-48bf-abe6-6a561d26ebb7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.762035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 117/233] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:22 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:22.769103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:22.864618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:22.919531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8564be-8044-4b6e-8e34-b094283b858f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:22.960302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8fcde9-007e-467a-af50-e130e1013823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:22.961442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8fcde9-007e-467a-af50-e130e1013823 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:22.961922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8fcde9-007e-467a-af50-e130e1013823 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:22.962405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8fcde9-007e-467a-af50-e130e1013823 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/d06cea73fe1845a985688560b2baa9ee/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:22.964884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8fcde9-007e-467a-af50-e130e1013823 None admin] RBAC: Authorizing `identity:create_grant(project_id=434cda3cc623466c9163ef469e5b2e7d, user_id=d06cea73fe1845a985688560b2baa9ee, role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:22.966557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8fcde9-007e-467a-af50-e130e1013823 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:22.983384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 117/234] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:22 2022] PUT /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/d06cea73fe1845a985688560b2baa9ee/roles/9d68999390534c65b1cfc53d421f6412 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:24.256956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2e62de-401c-4194-887a-b710f0e60162 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:24.257916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2e62de-401c-4194-887a-b710f0e60162 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:24.258389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2e62de-401c-4194-887a-b710f0e60162 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:24.260198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 118/235] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:24.266268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dfbd4c-badb-402d-bf86-6ff44e0fa3e1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:24.267182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dfbd4c-badb-402d-bf86-6ff44e0fa3e1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:24.267651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7dfbd4c-badb-402d-bf86-6ff44e0fa3e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:24.296952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d7dfbd4c-badb-402d-bf86-6ff44e0fa3e1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:24.298823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d7dfbd4c-badb-402d-bf86-6ff44e0fa3e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:24.347215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa33222-3774-47a8-bd29-7194104bc088 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:24.347557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa33222-3774-47a8-bd29-7194104bc088 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:24.347766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa33222-3774-47a8-bd29-7194104bc088 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:24.349154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 119/236] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:24.354166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1482961-1f49-4a74-a68f-2b6f91796101 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:24.354510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1482961-1f49-4a74-a68f-2b6f91796101 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:24.354794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1482961-1f49-4a74-a68f-2b6f91796101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:24.374020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e1482961-1f49-4a74-a68f-2b6f91796101 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:24.375237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e1482961-1f49-4a74-a68f-2b6f91796101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:24.402459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 118/237] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:24.448060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 120/238] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:24.703099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7dfbd4c-badb-402d-bf86-6ff44e0fa3e1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:24.728716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1482961-1f49-4a74-a68f-2b6f91796101 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:24.777058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:24.846912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1482961-1f49-4a74-a68f-2b6f91796101 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:24.876493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:24.883185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7dfbd4c-badb-402d-bf86-6ff44e0fa3e1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:24.887037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:24.887968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:24.888448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:24.888917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:24.890246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:24.892008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:24.893803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 121/239] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:24.899277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:24.957006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:24.958028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:24.958506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:24.958971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:24.979520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:24.988738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:24.989647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:24.990426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 119/240] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:24.995869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:25.032883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d46cc3a-2d4b-44d3-89d4-a5d94fb79070 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:25.069367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:25.070341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.070819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.071286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.072156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:25.073786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:25.097435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 122/241] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:24 2022] POST /identity/v3/users => generated 293 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:40:25.199086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378d22d8-f345-4f71-b05a-4409128a3cc4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:25.213582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:25.249461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a75c1ad-9a91-48de-bbe5-4eff970a7ff5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:25.250083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75c1ad-9a91-48de-bbe5-4eff970a7ff5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.250897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75c1ad-9a91-48de-bbe5-4eff970a7ff5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.251168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75c1ad-9a91-48de-bbe5-4eff970a7ff5 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.251927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a75c1ad-9a91-48de-bbe5-4eff970a7ff5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:25.252250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a75c1ad-9a91-48de-bbe5-4eff970a7ff5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:25.253748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a75c1ad-9a91-48de-bbe5-4eff970a7ff5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:25.269802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 120/242] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:25.275885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:25.355044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:25.389274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e5f756-8f85-4b67-b5f0-6a52a53b7e83 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.389575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e5f756-8f85-4b67-b5f0-6a52a53b7e83 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.389821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e5f756-8f85-4b67-b5f0-6a52a53b7e83 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.397148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 121/243] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:25 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:25.402278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b9a51-487b-44c4-b11c-6ca7aecc5a35 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.402557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b9a51-487b-44c4-b11c-6ca7aecc5a35 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.402926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b9a51-487b-44c4-b11c-6ca7aecc5a35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.413524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc436b00-c98b-4f12-8309-5ab28258174f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:25.436483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e99b9a51-487b-44c4-b11c-6ca7aecc5a35 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:25.437672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e99b9a51-487b-44c4-b11c-6ca7aecc5a35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:25.453933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a5f7eb9-3c1f-4538-811a-dbaee7247b48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:25.454946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5f7eb9-3c1f-4538-811a-dbaee7247b48 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.455428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5f7eb9-3c1f-4538-811a-dbaee7247b48 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.455976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5f7eb9-3c1f-4538-811a-dbaee7247b48 None admin] PATH_INFO: `/v3/users/bc087f38c15d43449c5e0e2ff041f690` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.471776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5f7eb9-3c1f-4538-811a-dbaee7247b48 None admin] RBAC: Authorizing `identity:get_user(user_id=bc087f38c15d43449c5e0e2ff041f690)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:25.473198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a5f7eb9-3c1f-4538-811a-dbaee7247b48 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:25.474905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 123/244] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:25 2022] GET /identity/v3/users/bc087f38c15d43449c5e0e2ff041f690 => generated 297 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:25.477303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602fffb1-9708-42ab-902a-7756656d2376 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.480166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602fffb1-9708-42ab-902a-7756656d2376 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.480450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602fffb1-9708-42ab-902a-7756656d2376 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.481846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 124/245] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:25 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:25.482596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602fffb1-9708-42ab-902a-7756656d2376 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:25.537763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 122/246] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:25.539973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3a504c-0776-4938-b734-4bc1473bda33 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.540268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3a504c-0776-4938-b734-4bc1473bda33 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.540531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3a504c-0776-4938-b734-4bc1473bda33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.560026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ba3a504c-0776-4938-b734-4bc1473bda33 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:25.561360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ba3a504c-0776-4938-b734-4bc1473bda33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:25.600445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602fffb1-9708-42ab-902a-7756656d2376 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:25.612443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:25.640753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:25.641403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:25.641680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:25.642057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:25.643755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 123/247] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:25.723540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:25.724964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:25.828000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 125/248] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:25.834145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3a504c-0776-4938-b734-4bc1473bda33 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:25.869157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:25.876190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:25.885032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:26.026740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3a504c-0776-4938-b734-4bc1473bda33 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:26.061822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f578387-dc7e-490f-8e30-b9e63b55f702 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:26.087067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:26.087708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.088003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.088277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.091704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:26.093183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:26.095530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 124/249] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:25 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:26.106612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:26.128903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eef1568-d2df-48a2-80c1-801248f2a0cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:26.129540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eef1568-d2df-48a2-80c1-801248f2a0cf None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.129820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eef1568-d2df-48a2-80c1-801248f2a0cf None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.130089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eef1568-d2df-48a2-80c1-801248f2a0cf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.148137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eef1568-d2df-48a2-80c1-801248f2a0cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-6eef1568-d2df-48a2-80c1-801248f2a0cf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:26.153053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:26.154080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 126/250] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:26.155365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91381ba1-35bb-472e-843f-f2ea439ac19f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.156441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91381ba1-35bb-472e-843f-f2ea439ac19f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.156441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91381ba1-35bb-472e-843f-f2ea439ac19f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.162838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 127/251] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:26 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:26.163631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91381ba1-35bb-472e-843f-f2ea439ac19f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:26.230893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:26.236512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec7943-a195-4010-9c23-9090245e3314 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:26.264876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:26.284651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:26.285816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.286255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.286661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.301028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91381ba1-35bb-472e-843f-f2ea439ac19f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:26.301519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:26.307660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:26.308774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:26.309744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:26.312440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 125/252] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:26.313673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:26.350909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136c0688-543f-4551-ab3a-a09843bd58b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:26.351999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c0688-543f-4551-ab3a-a09843bd58b8 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.352449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c0688-543f-4551-ab3a-a09843bd58b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.353693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c0688-543f-4551-ab3a-a09843bd58b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.354578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136c0688-543f-4551-ab3a-a09843bd58b8 None admin] RBAC: Adding query filter params (role.id=aca3a5ddd1af426f83c15692fff0a29f, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=bc087f38c15d43449c5e0e2ff041f690) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:26.355083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136c0688-543f-4551-ab3a-a09843bd58b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:26.356715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136c0688-543f-4551-ab3a-a09843bd58b8 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:26.364825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 128/253] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:26 2022] GET /identity/v3/role_assignments?user.id=bc087f38c15d43449c5e0e2ff041f690&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=aca3a5ddd1af426f83c15692fff0a29f => generated 259 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:26.374646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ce4fbb-d6cb-4288-9487-23767e029183 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.374959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ce4fbb-d6cb-4288-9487-23767e029183 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.375192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ce4fbb-d6cb-4288-9487-23767e029183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.414575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-41ce4fbb-d6cb-4288-9487-23767e029183 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:26.415778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-41ce4fbb-d6cb-4288-9487-23767e029183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:26.495845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425eaa9e-5921-4ce6-beee-a18d60f2c4ec None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:26.571310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 129/254] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:26.571988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ce4fbb-d6cb-4288-9487-23767e029183 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:26.576893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:26.577518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.577789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.578052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.579433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:26.580096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:26.581556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:26.600547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 126/255] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:26.606661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:26.648306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:26.653635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:26.724458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ce4fbb-d6cb-4288-9487-23767e029183 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:26.811439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:26.811439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.811439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.811439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.811789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:26.811789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:26.813830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:26.817408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e2c3e0-c7b2-4529-b1ea-1f401c6a1aac None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:26.856975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 130/256] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:26.857896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:26.902813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:26.903447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:26.903718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:26.904002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] PATH_INFO: `/v3/users/d06cea73fe1845a985688560b2baa9ee` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:26.909223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] RBAC: Authorizing `identity:get_user(user_id=d06cea73fe1845a985688560b2baa9ee)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:26.910681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:26.916046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 127/257] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:26 2022] GET /identity/v3/users/d06cea73fe1845a985688560b2baa9ee => generated 283 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:26.942037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:27.037721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fb130-536e-4b73-8e40-26693588fcd5 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:27.066447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:27.085982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:27.086945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:27.087434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:27.087894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] PATH_INFO: `/v3/users/d50463a9b78b40338679be4183c85c5c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:27.092314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:27.093001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] RBAC: Authorizing `identity:get_user(user_id=d50463a9b78b40338679be4183c85c5c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:27.094807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:27.096477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 131/258] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:26 2022] GET /identity/v3/users/d50463a9b78b40338679be4183c85c5c => generated 297 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:27.097264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:27.110367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e960b37f-e14e-45f0-b5ec-ffc64d70cf35 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:27.176770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:27.177430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:27.177716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:27.177984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:27.286672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730b0634-52d6-4660-8297-bd27f470ca24 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:27.311795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:27.313345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:27.365276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:27.366580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:27.367050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:27.367525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:27.389544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:27.400411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:27.401311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:27.402080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 132/259] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:27.402901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:27.423439 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:27.442496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 128/260] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:40:27.443301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:27.524298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:27.576367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c605b-6aab-4402-baaa-2c518d16bcc2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:27.585913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4756da2-12f0-4ef5-825e-39e910606c09 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:27.616899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:27.617451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:27.617725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:27.617992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:27.619113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:27.620724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:27.622351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 133/261] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:27 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:27.623181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:27.668429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:27.669039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:27.669312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:27.669576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:27.670284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:27.670582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:27.671966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:27.692330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 129/262] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:27.697157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:27.794225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd3ff7e-91a6-4e4b-88b3-76cd246bb2a8 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:27.794877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:27.845570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:27.863956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4780756d-e7a7-4bd7-8279-13588f992719 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:27.876079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:27.876632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:27.876911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:27.877181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:27.884700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:27.886275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:27.888734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 134/263] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:27 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:27.889632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:27.901915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:27.902571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:27.902844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:27.903109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/d50463a9b78b40338679be4183c85c5c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:27.905579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] RBAC: Authorizing `identity:create_grant(project_id=434cda3cc623466c9163ef469e5b2e7d, user_id=d50463a9b78b40338679be4183c85c5c, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:27.907152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:27.927431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 130/264] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:27 2022] PUT /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/d50463a9b78b40338679be4183c85c5c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:27.928634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:28.089673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb78c86-861a-4949-8c92-5ba78dea8bdf None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:28.116370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477aa8b5-e28c-4d68-b80f-86b31c2158e5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:28.129923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:28.133697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:28.134343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.134616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.134945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] PATH_INFO: `/v3/users/d06cea73fe1845a985688560b2baa9ee` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.136100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] RBAC: Authorizing `identity:get_user(user_id=d06cea73fe1845a985688560b2baa9ee)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:28.137881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:28.139978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 135/265] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:27 2022] GET /identity/v3/users/d06cea73fe1845a985688560b2baa9ee => generated 283 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:28.145897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:28.166000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c6693f-98f4-468c-a065-818f29891549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:28.166876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c6693f-98f4-468c-a065-818f29891549 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.167155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c6693f-98f4-468c-a065-818f29891549 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.167429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c6693f-98f4-468c-a065-818f29891549 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.168375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c6693f-98f4-468c-a065-818f29891549 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=d06cea73fe1845a985688560b2baa9ee) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:28.168685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c6693f-98f4-468c-a065-818f29891549 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:28.170467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c6693f-98f4-468c-a065-818f29891549 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:28.178455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 131/266] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:27 2022] GET /identity/v3/role_assignments?user.id=d06cea73fe1845a985688560b2baa9ee&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:28.219590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:28.232064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6655bebd-0f93-41bc-990e-49277672f147 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.232371 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6655bebd-0f93-41bc-990e-49277672f147 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.232609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6655bebd-0f93-41bc-990e-49277672f147 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.234271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 132/267] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:28.240743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7333a7-d368-490e-916e-c1601d880967 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.241038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7333a7-d368-490e-916e-c1601d880967 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.241283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7333a7-d368-490e-916e-c1601d880967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.271801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81005449-93b9-462a-8cff-3561fdcbc154 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:28.286542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bf7333a7-d368-490e-916e-c1601d880967 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:28.287831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bf7333a7-d368-490e-916e-c1601d880967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:28.312599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:28.314028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.314350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.314624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.380869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 133/268] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:28.382994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd5d693-87fa-4b63-8eec-fd115e7bb9be None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.383517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd5d693-87fa-4b63-8eec-fd115e7bb9be None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.384085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd5d693-87fa-4b63-8eec-fd115e7bb9be None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.386398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 134/269] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:28.392545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b348e76-decc-49ee-9086-3e06fdeef99a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.393254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b348e76-decc-49ee-9086-3e06fdeef99a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.394179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b348e76-decc-49ee-9086-3e06fdeef99a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.406519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:28.408245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:28.416869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9b348e76-decc-49ee-9086-3e06fdeef99a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:28.418291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9b348e76-decc-49ee-9086-3e06fdeef99a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:28.468539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:28.495095 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 135/270] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:28.518608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 136/271] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:40:28.524539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b348e76-decc-49ee-9086-3e06fdeef99a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:28.560082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:28.650464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b348e76-decc-49ee-9086-3e06fdeef99a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:28.690340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:28.690964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.691237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.691501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.692623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:28.694125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:28.695984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 136/272] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:28 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:28.702020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:28.769082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:28.852776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659e4f9b-8cf3-4aad-9b22-37e5e6c24966 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:28.881599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5d4e8b-2399-4517-b497-053450b6de5b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:28.883142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:28.893440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:28.894046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.894321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.894586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.895427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] RBAC: Adding query filter params (role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=d06cea73fe1845a985688560b2baa9ee) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:28.895730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:28.897226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:28.904145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 137/273] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:28 2022] GET /identity/v3/role_assignments?user.id=d06cea73fe1845a985688560b2baa9ee&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=9d68999390534c65b1cfc53d421f6412 => generated 608 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:28.904705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:28.933385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:28.945296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:28.945943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:28.946215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:28.946479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:28.977542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:28.990872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:28.991823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:28.993091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 137/274] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:28.998559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:29.049543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f946a491-a904-4b67-8b27-f672c7f63c69 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:29.110599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:29.111266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.111540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.111807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.113912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:29.115376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:29.127096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 138/275] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:29.128126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:29.215193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8659dd4-23ce-4e03-93bd-215c99693029 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:29.255281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:29.255891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.256286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.256559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.257227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:29.257526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:29.259075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:29.262857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:29.271167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 138/276] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:29.275264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74f0853-6b7a-4be5-932c-63e1ed9f2c27 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:29.285727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:29.321354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:29.324790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510fa980-afdf-42f0-a675-b01e7428d3c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:29.325401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510fa980-afdf-42f0-a675-b01e7428d3c5 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.325673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510fa980-afdf-42f0-a675-b01e7428d3c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.325941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510fa980-afdf-42f0-a675-b01e7428d3c5 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.326582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510fa980-afdf-42f0-a675-b01e7428d3c5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:29.328839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510fa980-afdf-42f0-a675-b01e7428d3c5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:29.355473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 139/277] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:29 2022] POST /identity/v3/users => generated 293 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:29.357260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd30f219-08ee-477e-9a19-b1d09c89b06e None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.357556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd30f219-08ee-477e-9a19-b1d09c89b06e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.357828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd30f219-08ee-477e-9a19-b1d09c89b06e None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.359667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 140/278] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:29.369880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594ecfd1-20d2-45c6-867c-52fe4e53aae8 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.370465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594ecfd1-20d2-45c6-867c-52fe4e53aae8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.370837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594ecfd1-20d2-45c6-867c-52fe4e53aae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.398836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-594ecfd1-20d2-45c6-867c-52fe4e53aae8 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:29.400122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-594ecfd1-20d2-45c6-867c-52fe4e53aae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:29.423540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b867b25-86d3-4fe5-9345-e940e7d5d973 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:29.463786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:29.464739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.465192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.465717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] PATH_INFO: `/v3/users/a83b24683cda41409690288238cefca4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.481268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] RBAC: Authorizing `identity:get_user(user_id=a83b24683cda41409690288238cefca4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:29.483084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:29.486238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 139/279] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:29 2022] GET /identity/v3/users/a83b24683cda41409690288238cefca4 => generated 293 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:29.494965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 141/280] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:29.496728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:29.590469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:29.612497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b576f29-a6c5-4c5c-8684-f5cec43720d4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:29.648964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:29.649963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.650437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.650901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.651870 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, user.id=a83b24683cda41409690288238cefca4, scope.system=all) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:29.652399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:29.654035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:29.660059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 140/281] 10.222.0.42 () {56 vars in 1308 bytes} [Fri Sep 16 01:40:29 2022] GET /identity/v3/role_assignments?user.id=a83b24683cda41409690288238cefca4&scope.system=all&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 226 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:29.771505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594ecfd1-20d2-45c6-867c-52fe4e53aae8 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:29.859373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:29.890026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594ecfd1-20d2-45c6-867c-52fe4e53aae8 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:29.929571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:29.930217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:29.930496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:29.930766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:29.943806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:29.948719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:29.949577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:29.950367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 142/282] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:29 2022] GET /identity/v3/roles/reader => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:29.958019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:30.113966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42762925-17b1-4687-96af-949efcebd3b1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:30.157629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:30.158474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:30.158892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:30.159298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:30.160147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:30.160580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:30.162075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:30.170963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 141/283] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:29 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:30.176917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:30.335503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0f0c58-c656-4195-b8d3-81d9e85b4128 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:30.387965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:30.388504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:30.388778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:30.389042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] PATH_INFO: `/v3/users/bc087f38c15d43449c5e0e2ff041f690` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:30.390173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] RBAC: Authorizing `identity:get_user(user_id=bc087f38c15d43449c5e0e2ff041f690)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:30.392627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:30.394523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 143/284] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:30 2022] GET /identity/v3/users/bc087f38c15d43449c5e0e2ff041f690 => generated 297 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:30.399707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:30.418658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:30.552112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240928a2-869e-48f7-b13b-a8d3245c3793 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:30.600754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6274a0b2-c25b-42b4-861a-ebe4095fe536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:30.601377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6274a0b2-c25b-42b4-861a-ebe4095fe536 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:30.601652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6274a0b2-c25b-42b4-861a-ebe4095fe536 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:30.601913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6274a0b2-c25b-42b4-861a-ebe4095fe536 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:30.604474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6274a0b2-c25b-42b4-861a-ebe4095fe536 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:30.605910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6274a0b2-c25b-42b4-861a-ebe4095fe536 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:30.607518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 142/285] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:30 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:30.613411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:30.617252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:30.742553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e95a74-dbf3-4a7c-a348-50b5e16f8535 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:30.791103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5061987-e765-4099-bea4-0fa84b3a561c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:30.791752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5061987-e765-4099-bea4-0fa84b3a561c None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:30.792062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5061987-e765-4099-bea4-0fa84b3a561c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:30.792330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5061987-e765-4099-bea4-0fa84b3a561c None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/bc087f38c15d43449c5e0e2ff041f690/roles/aca3a5ddd1af426f83c15692fff0a29f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:30.799183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5061987-e765-4099-bea4-0fa84b3a561c None admin] RBAC: Authorizing `identity:create_grant(project_id=434cda3cc623466c9163ef469e5b2e7d, user_id=bc087f38c15d43449c5e0e2ff041f690, role_id=aca3a5ddd1af426f83c15692fff0a29f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:30.799183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5061987-e765-4099-bea4-0fa84b3a561c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:30.820335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 144/286] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:40:30 2022] PUT /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d/users/bc087f38c15d43449c5e0e2ff041f690/roles/aca3a5ddd1af426f83c15692fff0a29f => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:30.947762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84b224-67df-48ea-b0e4-55f5f913d331 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:30.948113 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84b224-67df-48ea-b0e4-55f5f913d331 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:30.948343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84b224-67df-48ea-b0e4-55f5f913d331 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:30.949692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 143/287] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:30.962907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6a260-1e11-407f-b3d1-2946b69c17c1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:30.963219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6a260-1e11-407f-b3d1-2946b69c17c1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:30.963454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf6a260-1e11-407f-b3d1-2946b69c17c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:30.993233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8bf6a260-1e11-407f-b3d1-2946b69c17c1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:30.994393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8bf6a260-1e11-407f-b3d1-2946b69c17c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:31.081886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:31.093173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 145/288] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:31.473107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84b224-67df-48ea-b0e4-55f5f913d331 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:31.581152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:31.613396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84b224-67df-48ea-b0e4-55f5f913d331 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:31.648462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:31.649428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:31.649924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:31.650385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:31.661564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:31.667223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:31.668116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:31.668904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:31.671488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 144/289] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:31.677892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf6a260-1e11-407f-b3d1-2946b69c17c1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:31.782142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf6a260-1e11-407f-b3d1-2946b69c17c1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:31.818418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:31.819041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:31.819318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:31.819586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:31.820285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:31.820582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:31.821956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:31.831319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 146/290] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:31.837090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:31.966871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad98568-01ae-457f-9a10-0c4e622f5d75 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:32.003282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13f1a60-d3ca-4afd-9e5e-e244b972eae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:32.003912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13f1a60-d3ca-4afd-9e5e-e244b972eae4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.004215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13f1a60-d3ca-4afd-9e5e-e244b972eae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.004487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13f1a60-d3ca-4afd-9e5e-e244b972eae4 None admin] PATH_INFO: `/v3/users/d50463a9b78b40338679be4183c85c5c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.005622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13f1a60-d3ca-4afd-9e5e-e244b972eae4 None admin] RBAC: Authorizing `identity:get_user(user_id=d50463a9b78b40338679be4183c85c5c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:32.007071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13f1a60-d3ca-4afd-9e5e-e244b972eae4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:32.008676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 145/291] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:31 2022] GET /identity/v3/users/d50463a9b78b40338679be4183c85c5c => generated 297 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:32.014514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:32.085304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:32.123442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9ab2c-3aa3-49d8-a72f-c7a174af5085 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:32.163548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:32.164437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.164858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.165295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.236894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:32.238580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:32.257753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c2850a-8d19-4986-8c03-e47dd5502972 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.258252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c2850a-8d19-4986-8c03-e47dd5502972 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.258692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c2850a-8d19-4986-8c03-e47dd5502972 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.260477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 146/292] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:32.265518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ec8cef-a4f1-48fc-93b0-8832641335d4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.266091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ec8cef-a4f1-48fc-93b0-8832641335d4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.266748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ec8cef-a4f1-48fc-93b0-8832641335d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.279397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:32.304514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:32.305136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-15ec8cef-a4f1-48fc-93b0-8832641335d4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:32.307356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-15ec8cef-a4f1-48fc-93b0-8832641335d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:32.353546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 147/293] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:32.359988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:32.433604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 147/294] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:32.434952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649851e-b17e-46a9-b525-e5165d85a2eb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.435236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649851e-b17e-46a9-b525-e5165d85a2eb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.435622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649851e-b17e-46a9-b525-e5165d85a2eb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.437164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 148/295] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:32.442863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db8bcbe-09eb-452f-8491-4c8267bbad47 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.444987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db8bcbe-09eb-452f-8491-4c8267bbad47 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.445390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db8bcbe-09eb-452f-8491-4c8267bbad47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.480714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6db8bcbe-09eb-452f-8491-4c8267bbad47 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:32.481906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6db8bcbe-09eb-452f-8491-4c8267bbad47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:32.484996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d04d99-d17c-4665-81ba-1590c04c54c3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:32.552685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:32.553586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.559344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.559844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.561362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:32.563872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:32.565822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 148/296] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:32 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:32.580401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:32.603964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 149/297] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:32.625409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:32.680443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6d288f-2bdb-4e96-8129-aa59f8a925a0 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.681216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6d288f-2bdb-4e96-8129-aa59f8a925a0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.681723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6d288f-2bdb-4e96-8129-aa59f8a925a0 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.684734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 150/298] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:32.689496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5bcb5-6c44-4483-b093-ef1ba80cacf2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.690000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5bcb5-6c44-4483-b093-ef1ba80cacf2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.690538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5bcb5-6c44-4483-b093-ef1ba80cacf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.716350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3af5bcb5-6c44-4483-b093-ef1ba80cacf2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:32.720999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3af5bcb5-6c44-4483-b093-ef1ba80cacf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:32.742447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b8bca2-268c-4e86-a4a9-16b6ebc10066 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:32.816655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:32.817253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:32.818778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:32.819298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:32.819790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:32.820738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=d50463a9b78b40338679be4183c85c5c) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:32.821346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:32.822952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:32.829982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 151/299] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:32.829982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5bcb5-6c44-4483-b093-ef1ba80cacf2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:32.840174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 149/300] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:32 2022] GET /identity/v3/role_assignments?user.id=d50463a9b78b40338679be4183c85c5c&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 608 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:32.901647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:32.959055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5bcb5-6c44-4483-b093-ef1ba80cacf2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.029299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.029949 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.030227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.030495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.031858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.035561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.037553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 152/301] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:33.043127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.061296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:33.076917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a6653b-5601-4f22-83fd-718c68a2ae77 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.122710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.123493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.123774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.124204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] PATH_INFO: `/v3/roles/member` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.125596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:33.140163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:33.148634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:33.149582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:33.150489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 150/302] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:33.151698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.169444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5e835e-b55b-4152-b15a-223e8c663f67 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.220269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.220881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.221155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.221418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] PATH_INFO: `/v3/groups` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.222385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.223483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.241466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 153/303] 10.222.0.42 () {60 vars in 1134 bytes} [Fri Sep 16 01:40:33 2022] POST /identity/v3/groups => generated 229 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:33.242707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.284663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39261ed9-7a55-4311-b91e-9463854bdabf None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.325046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.325474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.325756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.326027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.339761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:33.345755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:33.346708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:33.347470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 151/304] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:33.352811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.365824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:33.394112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61dd80e-38e9-44f5-a67b-783fe2252ec2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.429980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:33.455409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.456318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.456766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.457195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.457982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:33.458178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.459581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.469376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 154/305] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:33.475492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.489679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d511210-41df-49ed-9077-a115e0ebb32d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.528964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.529575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.529851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.530121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.530786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:33.531703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.533617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.543408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 152/306] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:33.548906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.602284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e1f075-4519-4458-a54a-c242f5fecf52 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.629813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:33.642501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea86c41f-833b-480f-8708-fcfbdb2e8c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.643151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86c41f-833b-480f-8708-fcfbdb2e8c62 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.643429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86c41f-833b-480f-8708-fcfbdb2e8c62 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.643689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86c41f-833b-480f-8708-fcfbdb2e8c62 None admin] PATH_INFO: `/v3/users/a83b24683cda41409690288238cefca4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.644860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea86c41f-833b-480f-8708-fcfbdb2e8c62 None admin] RBAC: Authorizing `identity:get_user(user_id=a83b24683cda41409690288238cefca4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.646304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea86c41f-833b-480f-8708-fcfbdb2e8c62 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.648102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 155/307] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:33 2022] GET /identity/v3/users/a83b24683cda41409690288238cefca4 => generated 293 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:33.649024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1504a-2bba-41f8-bb85-9581ea9c0a06 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.649306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1504a-2bba-41f8-bb85-9581ea9c0a06 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.649579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1504a-2bba-41f8-bb85-9581ea9c0a06 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.650906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 156/308] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:33 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:33.656097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e373fd10-659a-473e-9a52-3b23d035b3fc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.657122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e373fd10-659a-473e-9a52-3b23d035b3fc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.657569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e373fd10-659a-473e-9a52-3b23d035b3fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.670696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a30cd63-c9f3-433c-8eb9-8f1301330d11 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.681690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e373fd10-659a-473e-9a52-3b23d035b3fc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:33.683430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e373fd10-659a-473e-9a52-3b23d035b3fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:33.708378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.709051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.709334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.709611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] PATH_INFO: `/v3/users/d21c8e6cc861479d865cf99f57922aee` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.723813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] RBAC: Authorizing `identity:get_user(user_id=d21c8e6cc861479d865cf99f57922aee)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.725283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.726923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 153/309] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:33 2022] GET /identity/v3/users/d21c8e6cc861479d865cf99f57922aee => generated 293 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:33.731681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.740098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:33.762055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 157/310] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:33.763131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e373fd10-659a-473e-9a52-3b23d035b3fc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:33.855741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8318a7dd-918c-4a90-baa3-ced86b0e3f2b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.884111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:33.893663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.894333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.894614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.894883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] PATH_INFO: `/v3/system/users/a83b24683cda41409690288238cefca4/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.896339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=a83b24683cda41409690288238cefca4, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.897705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.905571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e373fd10-659a-473e-9a52-3b23d035b3fc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:33.908758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 154/311] 10.222.0.42 () {58 vars in 1334 bytes} [Fri Sep 16 01:40:33 2022] PUT /identity/v3/system/users/a83b24683cda41409690288238cefca4/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:33.953893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:33.954489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:33.954776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:33.955049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:33.955790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] RBAC: Adding query filter params (role.id=aca3a5ddd1af426f83c15692fff0a29f, user.id=d21c8e6cc861479d865cf99f57922aee, scope.system=all) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:33.956131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:33.957653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:33.963987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 158/312] 10.222.0.42 () {56 vars in 1308 bytes} [Fri Sep 16 01:40:33 2022] GET /identity/v3/role_assignments?user.id=d21c8e6cc861479d865cf99f57922aee&scope.system=all&role.id=aca3a5ddd1af426f83c15692fff0a29f => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:34.064982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:34.181804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86c0ace-d002-486e-9994-45dce281cb43 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:34.219190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:34.220315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:34.220813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:34.221275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:34.232327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:34.238111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:34.238974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:34.239795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:34.242467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 155/313] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:34.247929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:34.297162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:34.353200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6960e1c-c79c-4300-8758-42a28a78d56a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:34.390066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:34.390882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:34.391213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:34.391434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:34.392219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:34.392534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:34.393907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:34.402550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 159/314] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:34.408283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:34.464686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:34.520527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89524871-e7ed-435d-a8ef-b2ce2d0900ab None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:34.555401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:34.556067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:34.556350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:34.556620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] PATH_INFO: `/v3/users/bc087f38c15d43449c5e0e2ff041f690` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:34.557733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] RBAC: Authorizing `identity:get_user(user_id=bc087f38c15d43449c5e0e2ff041f690)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:34.559169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:34.560765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 156/315] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:34 2022] GET /identity/v3/users/bc087f38c15d43449c5e0e2ff041f690 => generated 297 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:34.566410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:34.670927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d736e1-6b26-4324-a277-816df8d9085b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:34.681001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:34.707491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:34.708131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:34.708413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:34.708686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:34.779577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:34.780982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:34.885180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 160/316] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:34.891509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:35.002331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af543e-5685-4cdd-a72e-90e7d28f9281 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:35.023890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:35.038825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89896b2a-9143-48fb-b1b6-e3960663e7d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:35.039392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89896b2a-9143-48fb-b1b6-e3960663e7d2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:35.039667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89896b2a-9143-48fb-b1b6-e3960663e7d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:35.039949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89896b2a-9143-48fb-b1b6-e3960663e7d2 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:35.041231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89896b2a-9143-48fb-b1b6-e3960663e7d2 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:35.042663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89896b2a-9143-48fb-b1b6-e3960663e7d2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:35.044379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 157/317] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:34 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:35.050893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:35.061041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:35.151379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa71f9d-d2cc-4e5c-82c0-3c6ad2ddd280 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:35.186272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5972f9e-0358-48d5-9907-e53e0593c65b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:35.186948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5972f9e-0358-48d5-9907-e53e0593c65b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:35.187304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5972f9e-0358-48d5-9907-e53e0593c65b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:35.187612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5972f9e-0358-48d5-9907-e53e0593c65b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:35.188407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5972f9e-0358-48d5-9907-e53e0593c65b None admin] RBAC: Adding query filter params (role.id=aca3a5ddd1af426f83c15692fff0a29f, scope.project.id=434cda3cc623466c9163ef469e5b2e7d, user.id=bc087f38c15d43449c5e0e2ff041f690) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:35.188717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5972f9e-0358-48d5-9907-e53e0593c65b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:35.190223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5972f9e-0358-48d5-9907-e53e0593c65b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:35.196161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 161/318] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:40:35 2022] GET /identity/v3/role_assignments?user.id=bc087f38c15d43449c5e0e2ff041f690&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=aca3a5ddd1af426f83c15692fff0a29f => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:35.522660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e91015-0cb8-43ea-ba81-ea284010675f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:35.523032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e91015-0cb8-43ea-ba81-ea284010675f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:35.523207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e91015-0cb8-43ea-ba81-ea284010675f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:35.524704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 158/319] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:35.529611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529f26f8-85ff-40f9-8d5f-85606d59d787 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:35.529970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529f26f8-85ff-40f9-8d5f-85606d59d787 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:35.530309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529f26f8-85ff-40f9-8d5f-85606d59d787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:35.548411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-529f26f8-85ff-40f9-8d5f-85606d59d787 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:35.549573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-529f26f8-85ff-40f9-8d5f-85606d59d787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:35.617500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 162/320] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:35.889049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e91015-0cb8-43ea-ba81-ea284010675f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:35.992940 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e91015-0cb8-43ea-ba81-ea284010675f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:36.025675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e33f97-c3bf-4a09-a65d-2e5c0192e483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:36.026292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e33f97-c3bf-4a09-a65d-2e5c0192e483 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.026569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e33f97-c3bf-4a09-a65d-2e5c0192e483 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.026834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e33f97-c3bf-4a09-a65d-2e5c0192e483 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.027912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e33f97-c3bf-4a09-a65d-2e5c0192e483 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:36.029351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e33f97-c3bf-4a09-a65d-2e5c0192e483 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:36.031099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 159/321] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:36.037046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529f26f8-85ff-40f9-8d5f-85606d59d787 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:36.039184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:36.142844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529f26f8-85ff-40f9-8d5f-85606d59d787 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:36.180244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cf751c-9b9b-415a-87ec-3dde9bebaca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:36.180852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cf751c-9b9b-415a-87ec-3dde9bebaca1 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.181127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cf751c-9b9b-415a-87ec-3dde9bebaca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.181391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cf751c-9b9b-415a-87ec-3dde9bebaca1 None admin] PATH_INFO: `/v3/groups` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.182018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cf751c-9b9b-415a-87ec-3dde9bebaca1 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:36.183449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cf751c-9b9b-415a-87ec-3dde9bebaca1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:36.199401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 163/322] 10.222.0.42 () {60 vars in 1134 bytes} [Fri Sep 16 01:40:36 2022] POST /identity/v3/groups => generated 226 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:40:36.251452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d53bd2-6a87-4a03-a5d1-59189ff7ad53 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.251794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d53bd2-6a87-4a03-a5d1-59189ff7ad53 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.252056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d53bd2-6a87-4a03-a5d1-59189ff7ad53 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.253485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 160/323] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:36.259048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63118ba3-18e3-4a3d-a61f-3ac3a50852b6 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.259386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63118ba3-18e3-4a3d-a61f-3ac3a50852b6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.259484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cde5ee5-227a-4bb4-9e5d-1d65c1f46f79 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.259658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63118ba3-18e3-4a3d-a61f-3ac3a50852b6 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.259751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cde5ee5-227a-4bb4-9e5d-1d65c1f46f79 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.259897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cde5ee5-227a-4bb4-9e5d-1d65c1f46f79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.261388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 164/324] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:36.267183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d6a079-fbf6-47f0-a95d-bd4b84ea8efc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.267975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d6a079-fbf6-47f0-a95d-bd4b84ea8efc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.267975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d6a079-fbf6-47f0-a95d-bd4b84ea8efc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.282550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0cde5ee5-227a-4bb4-9e5d-1d65c1f46f79 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:36.283796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0cde5ee5-227a-4bb4-9e5d-1d65c1f46f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:36.287715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a8d6a079-fbf6-47f0-a95d-bd4b84ea8efc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:36.289103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a8d6a079-fbf6-47f0-a95d-bd4b84ea8efc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:36.349743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:36.353267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:36.366434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 165/325] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:36.371512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 161/326] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:36.639356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d6a079-fbf6-47f0-a95d-bd4b84ea8efc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:36.642820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cde5ee5-227a-4bb4-9e5d-1d65c1f46f79 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:36.777884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cde5ee5-227a-4bb4-9e5d-1d65c1f46f79 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:36.778207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d6a079-fbf6-47f0-a95d-bd4b84ea8efc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:36.830565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:36.831194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.831474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.831749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.832587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:36.833323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:36.833700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:36.834147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:36.845015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:36.846708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:36.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:36.851687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:36.852506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:36.853110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:36.853888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:36.854655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:36.855238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 166/327] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:36 2022] GET /identity/v3/roles/reader => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:36.855305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 162/328] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:36.861990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:36.866559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:36.937165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:36.945537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:37.009017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6d1d56-a76b-4f1b-89cb-a2e2aee9a432 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:37.020238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a42666-fb41-44ff-b216-1171741f747f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:37.063162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:37.063837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:37.064131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:37.064595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:37.065283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:37.065583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:37.067117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:37.077163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 167/329] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:37.082407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:37.086690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:37.087317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:37.087594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:37.087862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:37.088558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:37.088858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:37.090242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:37.110578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 163/330] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:37.116220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:37.264092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dc5640-1e46-4a6f-aae2-d56c145dc347 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:37.275700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:37.306970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:37.307793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:37.308173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:37.308586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] PATH_INFO: `/v3/users/d21c8e6cc861479d865cf99f57922aee` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:37.309849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] RBAC: Authorizing `identity:get_user(user_id=d21c8e6cc861479d865cf99f57922aee)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:37.311378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad00ab4a-5257-4e1a-86b2-f3d26d111339 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:37.311498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:37.313638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 164/331] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:37 2022] GET /identity/v3/users/d21c8e6cc861479d865cf99f57922aee => generated 293 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:37.319011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:37.350272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:37.404636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:37.405638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:37.406176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:37.406919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] PATH_INFO: `/v3/users/a83b24683cda41409690288238cefca4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:37.408377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] RBAC: Authorizing `identity:get_user(user_id=a83b24683cda41409690288238cefca4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:37.410044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:37.412175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 168/332] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:37 2022] GET /identity/v3/users/a83b24683cda41409690288238cefca4 => generated 293 bytes in 330 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:37.417831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:37.448010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34675f2b-7b57-4b96-958f-c303ca74c195 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:37.490719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4f7c9a-190a-4eac-a87f-39fc077f5299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:37.491445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f7c9a-190a-4eac-a87f-39fc077f5299 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:37.491722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f7c9a-190a-4eac-a87f-39fc077f5299 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:37.492038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4f7c9a-190a-4eac-a87f-39fc077f5299 None admin] PATH_INFO: `/v3/system/users/d21c8e6cc861479d865cf99f57922aee/roles/aca3a5ddd1af426f83c15692fff0a29f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:37.493469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4f7c9a-190a-4eac-a87f-39fc077f5299 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=d21c8e6cc861479d865cf99f57922aee, role_id=aca3a5ddd1af426f83c15692fff0a29f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:37.494999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4f7c9a-190a-4eac-a87f-39fc077f5299 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:37.507994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 165/333] 10.222.0.42 () {58 vars in 1334 bytes} [Fri Sep 16 01:40:37 2022] PUT /identity/v3/system/users/d21c8e6cc861479d865cf99f57922aee/roles/aca3a5ddd1af426f83c15692fff0a29f => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:37.615550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a2fd43-f25f-4a7e-8e9a-ead1140dcfa9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:37.663557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846394a6-b484-4481-a6a2-7b49c6dcd5b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:37.664186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846394a6-b484-4481-a6a2-7b49c6dcd5b2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:37.664460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846394a6-b484-4481-a6a2-7b49c6dcd5b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:37.664728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846394a6-b484-4481-a6a2-7b49c6dcd5b2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:37.665476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846394a6-b484-4481-a6a2-7b49c6dcd5b2 None admin] RBAC: Adding query filter params (role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, user.id=a83b24683cda41409690288238cefca4, scope.system=all) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:37.665769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846394a6-b484-4481-a6a2-7b49c6dcd5b2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:37.667208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846394a6-b484-4481-a6a2-7b49c6dcd5b2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:37.684521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 169/334] 10.222.0.42 () {56 vars in 1308 bytes} [Fri Sep 16 01:40:37 2022] GET /identity/v3/role_assignments?user.id=a83b24683cda41409690288238cefca4&scope.system=all&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 510 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:38.976237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7e14f-1c5a-4113-bae8-90cdf2b5dd10 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:38.977342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7e14f-1c5a-4113-bae8-90cdf2b5dd10 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:38.977848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7e14f-1c5a-4113-bae8-90cdf2b5dd10 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:38.979870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 166/335] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:38.987276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7463ea9-a946-4cba-8fbd-f53cd89c9756 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:38.988220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7463ea9-a946-4cba-8fbd-f53cd89c9756 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:38.988705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7463ea9-a946-4cba-8fbd-f53cd89c9756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:39.017333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d7463ea9-a946-4cba-8fbd-f53cd89c9756 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:39.018847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d7463ea9-a946-4cba-8fbd-f53cd89c9756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:39.064952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:39.107552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 170/336] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:39.121167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9819ebd-2180-4049-9767-c76cc5e7df2c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:39.121781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9819ebd-2180-4049-9767-c76cc5e7df2c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:39.122246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9819ebd-2180-4049-9767-c76cc5e7df2c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:39.123759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 167/337] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:39.129367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453dc2dc-3ec8-480f-9ae6-6ad62ac778b2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:39.129904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453dc2dc-3ec8-480f-9ae6-6ad62ac778b2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:39.130381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453dc2dc-3ec8-480f-9ae6-6ad62ac778b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:39.157323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-453dc2dc-3ec8-480f-9ae6-6ad62ac778b2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:39.158836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-453dc2dc-3ec8-480f-9ae6-6ad62ac778b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:39.242169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 171/338] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:39.664761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9819ebd-2180-4049-9767-c76cc5e7df2c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:39.724740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:39.803324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453dc2dc-3ec8-480f-9ae6-6ad62ac778b2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:39.826943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9819ebd-2180-4049-9767-c76cc5e7df2c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:39.909578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb4ad4e-190b-413d-8e1c-0ab2b8b8dde0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:39.910217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb4ad4e-190b-413d-8e1c-0ab2b8b8dde0 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:39.910491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb4ad4e-190b-413d-8e1c-0ab2b8b8dde0 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:39.910755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb4ad4e-190b-413d-8e1c-0ab2b8b8dde0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:39.942533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:39.948559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb4ad4e-190b-413d-8e1c-0ab2b8b8dde0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:39.950591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453dc2dc-3ec8-480f-9ae6-6ad62ac778b2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-dbb4ad4e-190b-413d-8e1c-0ab2b8b8dde0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:39.955022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:39.955911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:39.956741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 168/339] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:39.957775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d54dbcf-bff4-49b3-9a21-3dd0236894ce None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:39.958272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d54dbcf-bff4-49b3-9a21-3dd0236894ce None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:39.958662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d54dbcf-bff4-49b3-9a21-3dd0236894ce None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:39.960330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 169/340] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:39.964979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d54dbcf-bff4-49b3-9a21-3dd0236894ce None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:39.997194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2be4444-c9ff-4a0f-8de2-09885a42e886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:39.997841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be4444-c9ff-4a0f-8de2-09885a42e886 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:39.998174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be4444-c9ff-4a0f-8de2-09885a42e886 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:39.998450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2be4444-c9ff-4a0f-8de2-09885a42e886 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:40.011214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2be4444-c9ff-4a0f-8de2-09885a42e886 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d2be4444-c9ff-4a0f-8de2-09885a42e886 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:40.021456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:40.022375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:40.023234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 172/341] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:40.023972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bff42b-7848-4474-bdc6-ae466a0a28ea None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:40.024471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bff42b-7848-4474-bdc6-ae466a0a28ea None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:40.024569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bff42b-7848-4474-bdc6-ae466a0a28ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:40.045404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b2bff42b-7848-4474-bdc6-ae466a0a28ea None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:40.047152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b2bff42b-7848-4474-bdc6-ae466a0a28ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:40.151436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 173/342] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:40.152648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bff42b-7848-4474-bdc6-ae466a0a28ea None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:40.202514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:40.210191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d54dbcf-bff4-49b3-9a21-3dd0236894ce None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:40.298227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:40.303852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:40.304518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:40.304796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:40.305066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:40.305739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:40.306039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:40.307422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:40.316609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bff42b-7848-4474-bdc6-ae466a0a28ea None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:40.333920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 170/343] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 369 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:40.339440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:40.383973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:40.383973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:40.383973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:40.383973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:40.384368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:40.384368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:40.386844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:40.400966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 174/344] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:40.422124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:40.540936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249dbe00-49ca-473d-ac4e-798dd98048a2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:40.610996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:40.611610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:40.611891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:40.612188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:40.638637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca57e21-06b7-432d-afe7-cc2094dd9e3c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:40.677231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:40.711915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:40.712564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:40.712847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:40.713120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:40.743691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:40.745201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:40.757411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:40.838052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:40.839864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:40.959403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 171/345] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 620 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:40.960865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:40.962641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 175/346] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 542 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:40:40.964548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:41.064458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:41.097623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:41.142815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c09b97f-20da-4190-bc65-32d9309533fd None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:41.195891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf437fc-d82c-4e23-b343-0b57319bfa08 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:41.212437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:41.214454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.219562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.220427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] PATH_INFO: `/v3/roles/member` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.237017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:41.242640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:41.243620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:41.244475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 172/347] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:41.245589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:41.250743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1951bdc-a506-4f9c-874d-5b319c5377d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:41.251401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1951bdc-a506-4f9c-874d-5b319c5377d6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.251671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1951bdc-a506-4f9c-874d-5b319c5377d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.251980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1951bdc-a506-4f9c-874d-5b319c5377d6 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.253119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1951bdc-a506-4f9c-874d-5b319c5377d6 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:41.254546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1951bdc-a506-4f9c-874d-5b319c5377d6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:41.256470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 176/348] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:40 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:41.257711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85ab13-63ce-4be0-8f58-52433f6ae918 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.257984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85ab13-63ce-4be0-8f58-52433f6ae918 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.258301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85ab13-63ce-4be0-8f58-52433f6ae918 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.259836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 177/349] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:41.261143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f493e1d-7528-41f0-b01c-abfe10610c34 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.261419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f493e1d-7528-41f0-b01c-abfe10610c34 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.261979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f493e1d-7528-41f0-b01c-abfe10610c34 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.263366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 178/350] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:41.268824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f493e1d-7528-41f0-b01c-abfe10610c34 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:41.368683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f17b115-5542-47ff-a380-a45228cf1dba None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:41.398373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:41.401136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f493e1d-7528-41f0-b01c-abfe10610c34 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:41.417983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2313a658-0fe1-4db4-9d5b-6a702865ab10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:41.418646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2313a658-0fe1-4db4-9d5b-6a702865ab10 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.418929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2313a658-0fe1-4db4-9d5b-6a702865ab10 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.419399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2313a658-0fe1-4db4-9d5b-6a702865ab10 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.422921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2313a658-0fe1-4db4-9d5b-6a702865ab10 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:41.425970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2313a658-0fe1-4db4-9d5b-6a702865ab10 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:41.426551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 173/351] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:41 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:41.427804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a3d1a2-3cb8-4993-89ed-96da3dc60268 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.428135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a3d1a2-3cb8-4993-89ed-96da3dc60268 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.428407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a3d1a2-3cb8-4993-89ed-96da3dc60268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.435973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:41.451811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-59a3d1a2-3cb8-4993-89ed-96da3dc60268 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:41.453022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:41.453206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-59a3d1a2-3cb8-4993-89ed-96da3dc60268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:41.453628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.453899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.454165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.454828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:41.455122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:41.456529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:41.465827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 179/352] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:41.466487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:41.545689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 174/353] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:41.548357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d9c963-055b-4257-a09d-87a2d3c8e68d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.548650 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d9c963-055b-4257-a09d-87a2d3c8e68d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.548925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d9c963-055b-4257-a09d-87a2d3c8e68d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.567058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-55d9c963-055b-4257-a09d-87a2d3c8e68d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:41.568638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-55d9c963-055b-4257-a09d-87a2d3c8e68d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:41.605514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4b5d1c-2982-43b0-8bfc-a94edceda018 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:41.653974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:41.655609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.656807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.657279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.660783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 175/354] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:41.661187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d9c963-055b-4257-a09d-87a2d3c8e68d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:41.665432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:41.666613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:41.668603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 180/355] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:41 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:41.669548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:41.727489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:41.734549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:41.794981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e868f2-699f-4ac7-af09-35fe92f3e620 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:41.802204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d9c963-055b-4257-a09d-87a2d3c8e68d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:41.835365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:41.836001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.836434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.836713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.849507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1da7dd-6e46-46a5-bb1e-2528e5e5a07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:41.850454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1da7dd-6e46-46a5-bb1e-2528e5e5a07f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.851000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1da7dd-6e46-46a5-bb1e-2528e5e5a07f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.851467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1da7dd-6e46-46a5-bb1e-2528e5e5a07f None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.852830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1da7dd-6e46-46a5-bb1e-2528e5e5a07f None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:41.854472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1da7dd-6e46-46a5-bb1e-2528e5e5a07f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:41.858206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 176/356] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:41 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:41.863964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5aa8b8-7ac1-4795-8bec-63cc29b1e56f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:41.863964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5aa8b8-7ac1-4795-8bec-63cc29b1e56f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:41.863964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5aa8b8-7ac1-4795-8bec-63cc29b1e56f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:41.863964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 177/357] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:41.867890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5aa8b8-7ac1-4795-8bec-63cc29b1e56f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:41.930447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:41.931864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:41.987973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:42.000569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5aa8b8-7ac1-4795-8bec-63cc29b1e56f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.024030 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:42.045950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.047235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.047794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.048008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.048755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:42.049057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:42.050480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:42.055351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 181/358] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:42.056231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.057524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 178/359] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:41 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 260 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:42.058335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.194291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec971c2-494a-4c11-a5c8-ccd391432d68 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.213695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad6105d-f858-4fdf-854e-0aa9d2aa4d22 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.243366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ec3595-c28f-440b-86be-8ab21a3f2379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.243877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ec3595-c28f-440b-86be-8ab21a3f2379 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.244320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ec3595-c28f-440b-86be-8ab21a3f2379 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.244435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ec3595-c28f-440b-86be-8ab21a3f2379 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.246876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ec3595-c28f-440b-86be-8ab21a3f2379 None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:42.246876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ec3595-c28f-440b-86be-8ab21a3f2379 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:42.246876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ec3595-c28f-440b-86be-8ab21a3f2379 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:42.254674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 179/360] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=9d68999390534c65b1cfc53d421f6412 => generated 260 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:42.257095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63f437f-afe0-460b-a7de-3c86fe34a9f6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.257375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63f437f-afe0-460b-a7de-3c86fe34a9f6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.257642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63f437f-afe0-460b-a7de-3c86fe34a9f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.262203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.263248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.263736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.264218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.279369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b63f437f-afe0-460b-a7de-3c86fe34a9f6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:42.279839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:42.280654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b63f437f-afe0-460b-a7de-3c86fe34a9f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:42.287511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:42.288516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:42.289393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:42.292475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 182/361] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:42.293944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.308490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:42.359061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:42.368575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 180/362] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:42.369728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63f437f-afe0-460b-a7de-3c86fe34a9f6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.463028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdeca82d-0c62-44d2-8246-f4927a6d3271 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.529470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.530211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.530490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.531413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.541113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63f437f-afe0-460b-a7de-3c86fe34a9f6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.542924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:42.549907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:42.550790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 16 01:40:42.551569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:42.556378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 183/363] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/roles/reader => generated 84 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:42.557353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.581506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.582301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.582588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.582858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.584007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:42.585692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:42.589944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 181/364] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:42.591064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.643807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:42.703415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbec679-d0d1-4691-aadc-83523bc43a8c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.708544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:42.743622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.744198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.744484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.744757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.745432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:42.745733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:42.747114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:42.753768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4779222c-b6ea-4801-9cd0-cd22f5aadd7d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.757737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 182/365] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:42.758711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.813200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.813200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.813200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.813486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.814101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:42.814404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:42.816353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:42.826118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 184/366] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:42.826978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.903246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3fb857-fe06-4f15-afe7-ff8bd0edee94 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:42.916102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:42.951969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:42.952648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:42.954016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:42.954016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:42.958840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:42.958840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:42.959055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 183/367] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:42.959725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:42.965841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c5d2e6-1a97-48a0-a316-2f0149244cd4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:43.007058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:43.017749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:43.018683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:43.019166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:43.019687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:43.030624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:43.037845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:43.038738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:43.039541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:43.042225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 185/368] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:43.043351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:43.092655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7740f761-f74a-4289-bd7c-aacc97cdd578 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:43.138684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:43.139675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:43.140292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:43.140997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] PATH_INFO: `/v3/users/d21c8e6cc861479d865cf99f57922aee` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:43.143183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] RBAC: Authorizing `identity:get_user(user_id=d21c8e6cc861479d865cf99f57922aee)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:43.145336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:43.147838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 184/369] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:40:42 2022] GET /identity/v3/users/d21c8e6cc861479d865cf99f57922aee => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:43.148793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:43.174848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a98458e-52ee-4f8d-b272-46c61b5ecbd9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:43.208331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:43.214909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:43.215574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:43.215857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:43.216180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:43.295280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c09c0-df3d-47b4-8f6d-dea9b6c94303 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:43.297207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:43.324366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:43.326345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:43.352526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:43.353141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:43.353432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:43.354073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:43.354808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=434cda3cc623466c9163ef469e5b2e7d) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:43.355108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:43.356690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:43.366963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 185/370] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:43 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 260 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:43.367916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:43.520861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 186/371] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:43.521750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:43.581916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01cfcbe-42bd-4089-96a0-e568fcfe6a8a None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:43.605563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:43.630903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:43.631522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:43.631800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:43.632101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:43.632933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:43.633922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:43.636263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:43.650214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 186/372] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:43.651082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:43.670342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:43.686066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b2ee0-31e1-45d8-b0d4-508a6c95129f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:43.742358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:43.743266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:43.744261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:43.745152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:43.746702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] RBAC: Adding query filter params (role.id=aca3a5ddd1af426f83c15692fff0a29f, user.id=d21c8e6cc861479d865cf99f57922aee, scope.system=all) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:43.747689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:43.749302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:43.775739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 187/373] 10.222.0.42 () {56 vars in 1308 bytes} [Fri Sep 16 01:40:43 2022] GET /identity/v3/role_assignments?user.id=d21c8e6cc861479d865cf99f57922aee&scope.system=all&role.id=aca3a5ddd1af426f83c15692fff0a29f => generated 510 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:43.776834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:43.842105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0e83a4-e93d-4489-a3d4-74afa76f2072 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:43.933233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:43.934146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:43.934568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:43.935007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:43.937985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:43.939657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:43.942320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 187/374] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:43 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:43.949005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:44.018907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275d739d-bd7e-4a73-9980-e639fcb80e51 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:44.025039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:44.062792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:44.063429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:44.063881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:44.064205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:44.085121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:44.160095 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:44.161736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:44.162386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b99ad68-f7dc-431a-a620-eafc6e354ab5 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:44.236526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:44.237545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:44.238037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:44.238502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:44.240440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:44.242082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:44.243850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 188/375] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:43 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:44.253676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:44.354837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 188/376] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 578 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:44.364026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:44.454276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:44.467015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3f7187-a28f-4e2c-8a34-3620eb47033f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:44.472263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:44.540753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4d53b9-0e7d-4e98-baf8-1686da45a485 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:44.552046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:44.552663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:44.552939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:44.553206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:44.553939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=434cda3cc623466c9163ef469e5b2e7d) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:44.554241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:44.555662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:44.567584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 189/377] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:44 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=9d68999390534c65b1cfc53d421f6412 => generated 260 bytes in 315 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:44.591062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c31e0b-6d0f-46f8-9388-9b2726b8d2f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:44.591810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c31e0b-6d0f-46f8-9388-9b2726b8d2f9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:44.592188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c31e0b-6d0f-46f8-9388-9b2726b8d2f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:44.592461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c31e0b-6d0f-46f8-9388-9b2726b8d2f9 None admin] PATH_INFO: `/v3/projects/efec9f6da9c342038dd7466c677c23ad` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:44.594508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c31e0b-6d0f-46f8-9388-9b2726b8d2f9 None admin] RBAC: Authorizing `identity:get_project(project_id=efec9f6da9c342038dd7466c677c23ad)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:44.596002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c31e0b-6d0f-46f8-9388-9b2726b8d2f9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:44.598121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 189/378] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:44 2022] GET /identity/v3/projects/efec9f6da9c342038dd7466c677c23ad => generated 344 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:44.603893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:44.742487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a408bbdb-e937-4f39-b3d9-7b2b7b99fd81 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:44.757565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203d98e-a27c-47dc-bb4c-d78736dbeb84 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:44.757904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203d98e-a27c-47dc-bb4c-d78736dbeb84 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:44.758180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7203d98e-a27c-47dc-bb4c-d78736dbeb84 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:44.764005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 190/379] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:44.774905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66028bc-b3ea-4dbc-9c53-0dae791f32ad None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:44.775244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66028bc-b3ea-4dbc-9c53-0dae791f32ad None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:44.775459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c66028bc-b3ea-4dbc-9c53-0dae791f32ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:44.809409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c66028bc-b3ea-4dbc-9c53-0dae791f32ad None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:44.811005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c66028bc-b3ea-4dbc-9c53-0dae791f32ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:44.813984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:44.814675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:44.814946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:44.815258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] PATH_INFO: `/v3/groups/ee3deb356179452f8c2600b902d262d3` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:44.829181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] RBAC: Authorizing `identity:get_group(group_id=ee3deb356179452f8c2600b902d262d3)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:44.830652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:44.835471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 190/380] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:44 2022] GET /identity/v3/groups/ee3deb356179452f8c2600b902d262d3 => generated 229 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:44.839142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:44.908572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:44.939675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 191/381] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:44.997229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f129157-2507-4b46-904a-0755b4d210d4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:45.039157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d704e14-a402-436a-9174-f0fa79a2ca72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:45.039787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d704e14-a402-436a-9174-f0fa79a2ca72 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:45.040157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d704e14-a402-436a-9174-f0fa79a2ca72 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:45.040420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d704e14-a402-436a-9174-f0fa79a2ca72 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:45.041187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d704e14-a402-436a-9174-f0fa79a2ca72 None admin] RBAC: Adding query filter params (group.id=ee3deb356179452f8c2600b902d262d3, role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=efec9f6da9c342038dd7466c677c23ad) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:45.041467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d704e14-a402-436a-9174-f0fa79a2ca72 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:45.042981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d704e14-a402-436a-9174-f0fa79a2ca72 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:45.056223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 191/382] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:44 2022] GET /identity/v3/role_assignments?group.id=ee3deb356179452f8c2600b902d262d3&scope.project.id=efec9f6da9c342038dd7466c677c23ad&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 260 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:45.271836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821c22af-e63b-4596-8f54-ecb2afac3bfa None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:45.272212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821c22af-e63b-4596-8f54-ecb2afac3bfa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:45.272382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821c22af-e63b-4596-8f54-ecb2afac3bfa None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:45.273917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 192/383] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:45.279002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64df88f-8cdd-4535-94a3-30702bb3db52 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:45.279288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64df88f-8cdd-4535-94a3-30702bb3db52 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:45.279558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64df88f-8cdd-4535-94a3-30702bb3db52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:45.285581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821c22af-e63b-4596-8f54-ecb2afac3bfa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:45.304547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b64df88f-8cdd-4535-94a3-30702bb3db52 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:45.305756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b64df88f-8cdd-4535-94a3-30702bb3db52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:45.355237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:45.401709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 192/384] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:45.423689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821c22af-e63b-4596-8f54-ecb2afac3bfa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:45.459646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:45.460299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:45.460631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:45.460902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:45.472217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:45.477081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:45.477978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:45.478829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 193/385] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:45.485305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64df88f-8cdd-4535-94a3-30702bb3db52 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:45.514520 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:45.594725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64df88f-8cdd-4535-94a3-30702bb3db52 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:45.631433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:45.632119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:45.632400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:45.632669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:45.633417 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:45.633711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:45.635083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:45.644691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 193/386] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:45.650191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:45.758213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b96705-3f24-4ad4-9a37-ce510a1ffea2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:45.760674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:45.808617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:45.809255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:45.809528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:45.809792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:45.810901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:45.812697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:45.814335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 194/387] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:45 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:45.819851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:45.915730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:45.921219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:45.943078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5962fb-1128-4814-8f19-b40333769aea None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:46.036703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:46.037338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.037611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.037873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.042130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:46.046869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:46.049250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 194/388] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:45 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:46.056082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:46.070223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92431798-0e91-4ebd-99a8-c0035c146091 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:46.144505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:46.146723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.147009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.147280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.173003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:46.183406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:46.184379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:46.185145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:46.189702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 195/389] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:46.193633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:46.252865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:46.269566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4423ca55-45b9-4abc-9ed7-950c20d03a94 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:46.333743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee18f18-6613-47b1-bfbf-a43551ca1565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:46.334412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee18f18-6613-47b1-bfbf-a43551ca1565 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.334686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee18f18-6613-47b1-bfbf-a43551ca1565 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.334953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee18f18-6613-47b1-bfbf-a43551ca1565 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/groups/5241104f2c73496583e0e0acebfe3d01/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.338225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee18f18-6613-47b1-bfbf-a43551ca1565 None admin] RBAC: Authorizing `identity:create_grant(project_id=5d0bb683d54f429588aceddf17bb9ef8, group_id=5241104f2c73496583e0e0acebfe3d01, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:46.339756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee18f18-6613-47b1-bfbf-a43551ca1565 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:46.356300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 195/390] 10.222.0.42 () {58 vars in 1442 bytes} [Fri Sep 16 01:40:46 2022] PUT /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/groups/5241104f2c73496583e0e0acebfe3d01/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:46.357897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cca31b0-e21e-4e85-bdc3-d0c857dda956 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.358256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cca31b0-e21e-4e85-bdc3-d0c857dda956 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.358749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cca31b0-e21e-4e85-bdc3-d0c857dda956 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.364015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 196/391] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:46.367842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f6e766-89bd-49e9-9f88-50f532fd5755 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.368229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f6e766-89bd-49e9-9f88-50f532fd5755 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.368502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f6e766-89bd-49e9-9f88-50f532fd5755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.373842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef4aa3c-9373-48a3-b22a-e0e83fe30248 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:46.401856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e9f6e766-89bd-49e9-9f88-50f532fd5755 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:46.403978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e9f6e766-89bd-49e9-9f88-50f532fd5755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:46.417258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:46.417933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.418213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.418480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.419264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:46.419578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:46.424073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:46.426165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:46.443837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 196/392] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:46.461663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:46.554998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 197/393] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:46.616500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:46.623037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68009572-32b3-4442-9f49-d63412f4e7c8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:46.658773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:46.659702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.660178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.660653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.661943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:46.663502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:46.665603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 197/394] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:46 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:46.672280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f6e766-89bd-49e9-9f88-50f532fd5755 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:46.798261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f6e766-89bd-49e9-9f88-50f532fd5755 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:46.834130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:46.835070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:46.835546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:46.836028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:46.837412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:46.839063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:46.841222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 198/395] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:46 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:46.847675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:46.896527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:46.914451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:47.020259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d504730b-f41f-4161-a9d7-ddd078a82e49 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:47.021966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3591c381-d382-4a50-afb9-9f812badde41 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:47.087201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:47.087780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.088086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.088359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/groups/5241104f2c73496583e0e0acebfe3d01/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.091185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:47.091813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.092108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.092375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.096262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] RBAC: Authorizing `identity:create_grant(project_id=5d0bb683d54f429588aceddf17bb9ef8, group_id=5241104f2c73496583e0e0acebfe3d01, role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:47.097697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:47.118959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 198/396] 10.222.0.42 () {58 vars in 1442 bytes} [Fri Sep 16 01:40:46 2022] PUT /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8/groups/5241104f2c73496583e0e0acebfe3d01/roles/9d68999390534c65b1cfc53d421f6412 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:47.119572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:47.125447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:47.126374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:47.127259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 199/397] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:47.132958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:47.176337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:47.277931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ea0d75-b748-4378-acbe-ff4531c76425 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:47.335780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5a6838-8a69-4319-a5f0-02fcba5fb8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:47.336882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5a6838-8a69-4319-a5f0-02fcba5fb8cd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.337422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5a6838-8a69-4319-a5f0-02fcba5fb8cd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.337907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5a6838-8a69-4319-a5f0-02fcba5fb8cd None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.338827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5a6838-8a69-4319-a5f0-02fcba5fb8cd None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:47.339353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5a6838-8a69-4319-a5f0-02fcba5fb8cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:47.341076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5a6838-8a69-4319-a5f0-02fcba5fb8cd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:47.355104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 199/398] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:47.361506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:47.461312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:47.479850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da229b6-d068-46f1-b7a0-06ed3015ec0f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.480408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da229b6-d068-46f1-b7a0-06ed3015ec0f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.480516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da229b6-d068-46f1-b7a0-06ed3015ec0f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.481989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 200/399] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:47.489228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae10982-b33e-42d1-8725-07016f99a181 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.489847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae10982-b33e-42d1-8725-07016f99a181 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.490425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae10982-b33e-42d1-8725-07016f99a181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.493225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4eefa2-d995-4abc-a018-1646f792d6c2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:47.509732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fae10982-b33e-42d1-8725-07016f99a181 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:47.511250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fae10982-b33e-42d1-8725-07016f99a181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:47.563392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:47.564198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.564372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.564657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.565926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:47.567390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:47.569310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 200/400] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:47 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:47.578039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:47.595930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 201/401] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:47.699051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f422dcef-0246-406d-9cc5-ecbcb5fd5e23 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.699682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f422dcef-0246-406d-9cc5-ecbcb5fd5e23 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.700181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f422dcef-0246-406d-9cc5-ecbcb5fd5e23 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.701869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 202/402] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:47.707706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f1b962-1945-498b-bc38-766b9e9d0f18 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.708265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f1b962-1945-498b-bc38-766b9e9d0f18 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.708740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f1b962-1945-498b-bc38-766b9e9d0f18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.736190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-83f1b962-1945-498b-bc38-766b9e9d0f18 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:47.737391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-83f1b962-1945-498b-bc38-766b9e9d0f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:47.755605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1367ce-45d9-4f4f-8ead-9196eeadc9ec None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:47.760520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:47.827179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:47.828425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:47.828425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:47.828425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:47.830392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:47.831824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:47.836241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 201/403] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:47 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:47.840032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:47.853822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:47.871424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 203/404] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:47.920051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f1b962-1945-498b-bc38-766b9e9d0f18 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:48.017600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc72e9b8-1b1c-48a2-8fbd-dcdd38178551 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:48.084904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f1b962-1945-498b-bc38-766b9e9d0f18 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:48.086721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:48.088960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:48.089250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:48.089526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d/groups/5241104f2c73496583e0e0acebfe3d01/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:48.093708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] RBAC: Authorizing `identity:create_grant(project_id=434cda3cc623466c9163ef469e5b2e7d, group_id=5241104f2c73496583e0e0acebfe3d01, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:48.095363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:48.116906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 202/405] 10.222.0.42 () {58 vars in 1442 bytes} [Fri Sep 16 01:40:47 2022] PUT /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d/groups/5241104f2c73496583e0e0acebfe3d01/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:48.139196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:48.140452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:48.141035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:48.141565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:48.161076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:48.174169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:48.175127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:48.175916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:48.178715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 204/406] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:48.190696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:48.210992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:48.220107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:48.354882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:48.398683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc6051b-a6ed-4f1d-b7ae-ec3b8d37703c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:48.427972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0e8bbe-4232-4326-ae20-a3989a3c9a06 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:48.477710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:48.478336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:48.478612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:48.478885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:48.479566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:48.479862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:48.481298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:48.488385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:48.489035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:48.489297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:48.489560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:48.503547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 203/407] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:48.508679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:48.510192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:48.516202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:48.517086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:48.517885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:48.528108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 205/408] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:48.529436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:48.630569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd6d2cd-2cb1-4a54-a1cb-3580a5ff0314 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:48.663451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:48.676703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:48.677679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:48.678160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:48.678615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:48.680935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:48.682576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:48.687168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 204/409] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:48 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:48.691556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:48.693090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:48.780200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2317bf1-af1d-4cb6-bc2d-b1329102d516 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:48.857761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:48.858374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:48.859265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:48.859678 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:48.860668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:48.860969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:48.862346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:48.884646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 206/410] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:48.889741 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:48.908158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec0a31a-4be2-4979-aa5f-8f21b439d641 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:48.986522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:48.987080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:48.987354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:48.987620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:48.989625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:48.991060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:48.997743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 205/411] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:48 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:48.998587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:49.105619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c4c405-371a-45ca-9906-0f87d73d29ea None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:49.144166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:49.166525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:49.183486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:49.184138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:49.184414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:49.184680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] PATH_INFO: `/v3/groups/ee3deb356179452f8c2600b902d262d3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:49.185823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] RBAC: Authorizing `identity:get_group(group_id=ee3deb356179452f8c2600b902d262d3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:49.192211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:49.196610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 207/412] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:48 2022] GET /identity/v3/groups/ee3deb356179452f8c2600b902d262d3 => generated 229 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:49.199803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:49.223496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7de26d7-a108-4d7d-b42f-8d97075ca65e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:49.292351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:49.293089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:49.293368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:49.293632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d/groups/5241104f2c73496583e0e0acebfe3d01/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:49.296285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] RBAC: Authorizing `identity:create_grant(project_id=434cda3cc623466c9163ef469e5b2e7d, group_id=5241104f2c73496583e0e0acebfe3d01, role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:49.298118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:49.341162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 206/413] 10.222.0.42 () {58 vars in 1442 bytes} [Fri Sep 16 01:40:48 2022] PUT /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d/groups/5241104f2c73496583e0e0acebfe3d01/roles/9d68999390534c65b1cfc53d421f6412 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:49.369474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed02a90-ce3a-4dc7-8c90-4d0da318317c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:49.410484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e505f0f3-27b8-49b4-8558-ecc792062a51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:49.411118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e505f0f3-27b8-49b4-8558-ecc792062a51 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:49.411398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e505f0f3-27b8-49b4-8558-ecc792062a51 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:49.411670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e505f0f3-27b8-49b4-8558-ecc792062a51 None admin] PATH_INFO: `/v3/projects/efec9f6da9c342038dd7466c677c23ad` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:49.412978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e505f0f3-27b8-49b4-8558-ecc792062a51 None admin] RBAC: Authorizing `identity:get_project(project_id=efec9f6da9c342038dd7466c677c23ad)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:49.414734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e505f0f3-27b8-49b4-8558-ecc792062a51 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:49.416984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 208/414] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:49 2022] GET /identity/v3/projects/efec9f6da9c342038dd7466c677c23ad => generated 344 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:49.423426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:49.439755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7604f0cd-279b-4266-b655-b2477c89633f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:49.440086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7604f0cd-279b-4266-b655-b2477c89633f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:49.440340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7604f0cd-279b-4266-b655-b2477c89633f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:49.441914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 209/415] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:49.447569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773132f-818c-4405-8dc8-84d8c6ce24c5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:49.447856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773132f-818c-4405-8dc8-84d8c6ce24c5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:49.448162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b773132f-818c-4405-8dc8-84d8c6ce24c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:49.468675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b773132f-818c-4405-8dc8-84d8c6ce24c5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:49.470115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b773132f-818c-4405-8dc8-84d8c6ce24c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:49.553153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:49.560085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 210/416] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:49.615295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40db245f-be54-4a8e-a5bc-dad1918ecae7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:49.643984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:49.652678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf3d174-38d7-4944-a78b-0d8475d0e013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:49.653341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3d174-38d7-4944-a78b-0d8475d0e013 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:49.653619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3d174-38d7-4944-a78b-0d8475d0e013 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:49.653889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3d174-38d7-4944-a78b-0d8475d0e013 None admin] PATH_INFO: `/v3/projects/efec9f6da9c342038dd7466c677c23ad/groups/ee3deb356179452f8c2600b902d262d3/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:49.656555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf3d174-38d7-4944-a78b-0d8475d0e013 None admin] RBAC: Authorizing `identity:create_grant(project_id=efec9f6da9c342038dd7466c677c23ad, group_id=ee3deb356179452f8c2600b902d262d3, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:49.658161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf3d174-38d7-4944-a78b-0d8475d0e013 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:49.679134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 207/417] 10.222.0.42 () {58 vars in 1442 bytes} [Fri Sep 16 01:40:49 2022] PUT /identity/v3/projects/efec9f6da9c342038dd7466c677c23ad/groups/ee3deb356179452f8c2600b902d262d3/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:40:49.944237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773132f-818c-4405-8dc8-84d8c6ce24c5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:50.013811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21e496f-9210-4696-b4a5-1345dd0a33d3 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.014142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21e496f-9210-4696-b4a5-1345dd0a33d3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.014372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21e496f-9210-4696-b4a5-1345dd0a33d3 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.018579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 208/418] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:50.022587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bddaca4-939a-4724-9bfe-b134fc6f8e9f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.022873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bddaca4-939a-4724-9bfe-b134fc6f8e9f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.023144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bddaca4-939a-4724-9bfe-b134fc6f8e9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.060978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b773132f-818c-4405-8dc8-84d8c6ce24c5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:50.065005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7bddaca4-939a-4724-9bfe-b134fc6f8e9f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:50.066218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7bddaca4-939a-4724-9bfe-b134fc6f8e9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:50.098539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:50.099182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.099568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.099849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.111023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:50.115927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:50.116818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:50.118203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 211/419] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:50.122753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:50.179490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 209/420] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:50.252100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:50.321357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514054fb-5f0a-4a25-b454-3a67fb28791f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:50.374035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:50.374870 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.375314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.375584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.376271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:50.376567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:50.377944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:50.392853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 212/421] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:50.398542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bddaca4-939a-4724-9bfe-b134fc6f8e9f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:50.507840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bddaca4-939a-4724-9bfe-b134fc6f8e9f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:50.526316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:50.548001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:50.548621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.548900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.549170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.617018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:50.622527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:50.624017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:50.738776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 210/422] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:50.745066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:50.764941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:50.869932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcb3144-b15c-4435-937a-1bee41b64b98 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:50.873833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840d680e-2b02-4c65-ab74-cd1e72cc73a5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:50.905120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:50.908759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9649c2c3-87d8-44da-96ae-e99106c1b505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:50.909671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649c2c3-87d8-44da-96ae-e99106c1b505 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.910305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649c2c3-87d8-44da-96ae-e99106c1b505 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.910789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649c2c3-87d8-44da-96ae-e99106c1b505 None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.912155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9649c2c3-87d8-44da-96ae-e99106c1b505 None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:50.913784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9649c2c3-87d8-44da-96ae-e99106c1b505 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:50.915875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 211/423] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:50 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:50.917334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf3ea1-2be7-4c67-bce7-53b39d5cce8d None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.917818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf3ea1-2be7-4c67-bce7-53b39d5cce8d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.918285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcf3ea1-2be7-4c67-bce7-53b39d5cce8d None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.919677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 212/424] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:50.923883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcf3ea1-2be7-4c67-bce7-53b39d5cce8d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:50.942868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c58a29f-decf-4a57-b67e-3c1d9007ce71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:50.944585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c58a29f-decf-4a57-b67e-3c1d9007ce71 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.945070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c58a29f-decf-4a57-b67e-3c1d9007ce71 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.945536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c58a29f-decf-4a57-b67e-3c1d9007ce71 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.961288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c58a29f-decf-4a57-b67e-3c1d9007ce71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4c58a29f-decf-4a57-b67e-3c1d9007ce71 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:50.968928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:50.969798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:50.970600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:50.973322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 213/425] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:50.974332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeacc3-71a2-4d85-8876-2fb04db18adc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:50.974605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeacc3-71a2-4d85-8876-2fb04db18adc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:50.974869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbeacc3-71a2-4d85-8876-2fb04db18adc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:50.993553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2bbeacc3-71a2-4d85-8876-2fb04db18adc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:50.994719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2bbeacc3-71a2-4d85-8876-2fb04db18adc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:51.031868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:51.052874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcf3ea1-2be7-4c67-bce7-53b39d5cce8d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:51.072040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 214/426] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:51.072040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbeacc3-71a2-4d85-8876-2fb04db18adc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:51.123321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:51.123871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:51.124224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:51.124687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:51.127293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:51.128740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:51.132972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 213/427] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:50 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:51.141542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:51.235954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbeacc3-71a2-4d85-8876-2fb04db18adc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:51.239136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:51.299124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:51.299724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:51.300015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:51.300280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:51.300938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:51.301229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:51.303337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:51.321025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 215/428] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:51.325110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:51.341869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d5203f-8cd2-476e-acc1-7eaf75b303bd None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:51.411880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:51.412431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:51.412697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:51.413019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:51.413831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:51.414041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:51.415450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:51.425524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 214/429] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:51 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 611 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:51.426644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:51.464315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:51.549992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f35dec2-5f3b-4079-b4a6-523029378e60 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:51.582363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be67ee5-c4e9-4b3b-a040-67d21021670e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:51.626921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:51.627617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:51.627896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:51.628186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:51.644849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:51.650156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:51.651029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:51.651908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 215/430] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:51.654246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:51.655040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:51.655143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:51.656329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:51.660852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:51.708425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:51.738854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:51.742163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:51.758458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:51.786493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6dffc0c-8e65-4308-bcd0-f60862993b3c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:51.844040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:51.845527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:51.846905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:51.847522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:51.848556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:51.849075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:51.850677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:51.862204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 216/431] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:40:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:51.867141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 216/432] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 542 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:51.868336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:51.872836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:51.957779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:51.987921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbad915d-5a5d-4569-80e2-fa0a56ef05e9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:51.989327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee06d2a-afc3-4605-8c54-c280654ecf4b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.029980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda2c440-1d2c-4d4a-93ea-6396393174d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.030831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda2c440-1d2c-4d4a-93ea-6396393174d5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.031211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda2c440-1d2c-4d4a-93ea-6396393174d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.031643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda2c440-1d2c-4d4a-93ea-6396393174d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.039033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb8f82a-36b4-48b7-8e40-a90e6287286a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.039660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb8f82a-36b4-48b7-8e40-a90e6287286a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.039985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb8f82a-36b4-48b7-8e40-a90e6287286a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.040264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb8f82a-36b4-48b7-8e40-a90e6287286a None admin] PATH_INFO: `/v3/projects/5d0bb683d54f429588aceddf17bb9ef8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.041414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb8f82a-36b4-48b7-8e40-a90e6287286a None admin] RBAC: Authorizing `identity:get_project(project_id=5d0bb683d54f429588aceddf17bb9ef8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:52.042853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb8f82a-36b4-48b7-8e40-a90e6287286a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:52.044686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 217/433] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:51 2022] GET /identity/v3/projects/5d0bb683d54f429588aceddf17bb9ef8 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:52.045733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec0eda-ccb7-4bda-bcb0-1962cdb4322c None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.046050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec0eda-ccb7-4bda-bcb0-1962cdb4322c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.046320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec0eda-ccb7-4bda-bcb0-1962cdb4322c None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.046607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:52.050816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 218/434] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:52.052982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ec0eda-ccb7-4bda-bcb0-1962cdb4322c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.127372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda2c440-1d2c-4d4a-93ea-6396393174d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:52.128875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda2c440-1d2c-4d4a-93ea-6396393174d5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:52.188182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ec0eda-ccb7-4bda-bcb0-1962cdb4322c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.228152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8a726b-4bc4-440c-b248-4436e4ba4eba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.228773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8a726b-4bc4-440c-b248-4436e4ba4eba None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.229048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8a726b-4bc4-440c-b248-4436e4ba4eba None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.229313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8a726b-4bc4-440c-b248-4436e4ba4eba None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.230420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8a726b-4bc4-440c-b248-4436e4ba4eba None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:52.231847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8a726b-4bc4-440c-b248-4436e4ba4eba None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:52.233510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 219/435] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:52.234603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d52d0-4862-4c94-b779-78711b009210 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.234878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d52d0-4862-4c94-b779-78711b009210 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.235148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29d52d0-4862-4c94-b779-78711b009210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.241379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:52.241952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 217/436] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:52.243183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9533f9-05b0-4d5c-b40d-1d20c9acbb45 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.243463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9533f9-05b0-4d5c-b40d-1d20c9acbb45 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9533f9-05b0-4d5c-b40d-1d20c9acbb45 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.248553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 218/437] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:52.248553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9533f9-05b0-4d5c-b40d-1d20c9acbb45 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.254717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b29d52d0-4862-4c94-b779-78711b009210 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:52.255900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b29d52d0-4862-4c94-b779-78711b009210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:52.289511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:52.324745 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 220/438] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:52.325946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29d52d0-4862-4c94-b779-78711b009210 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.350825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9533f9-05b0-4d5c-b40d-1d20c9acbb45 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.384475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3001450-122f-45aa-8aef-ec9361802d99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.386154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3001450-122f-45aa-8aef-ec9361802d99 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.386154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3001450-122f-45aa-8aef-ec9361802d99 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.386154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3001450-122f-45aa-8aef-ec9361802d99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.387345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3001450-122f-45aa-8aef-ec9361802d99 None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=5d0bb683d54f429588aceddf17bb9ef8) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:52.387651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3001450-122f-45aa-8aef-ec9361802d99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:52.389230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3001450-122f-45aa-8aef-ec9361802d99 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:52.395798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 219/439] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=5d0bb683d54f429588aceddf17bb9ef8&role.id=9d68999390534c65b1cfc53d421f6412 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:52.396819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93d583-b1a4-41cc-bb87-2cafaa45ba00 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.397104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93d583-b1a4-41cc-bb87-2cafaa45ba00 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.397375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93d583-b1a4-41cc-bb87-2cafaa45ba00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.429179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6b93d583-b1a4-41cc-bb87-2cafaa45ba00 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:52.430925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6b93d583-b1a4-41cc-bb87-2cafaa45ba00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:52.451208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b29d52d0-4862-4c94-b779-78711b009210 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.487547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:52.494038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.495002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.495422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.495829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.497320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:52.498942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:52.501277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 221/440] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:52.507195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.526161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 220/441] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:52.595884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b93d583-b1a4-41cc-bb87-2cafaa45ba00 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.620248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3abfff-caae-4325-b2af-90c039ea0501 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.639041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:52.661755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.662389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.662645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.662909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.664697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:52.666426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:52.668640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 222/442] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:52.676877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.732330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b93d583-b1a4-41cc-bb87-2cafaa45ba00 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.768660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:52.784248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.784984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.785338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.785942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.802542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:52.809099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:52.809973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 221/443] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:52.812838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.817099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdf9882-2946-4878-9b06-f45a277c0160 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.855565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.856350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.856628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.856896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:52.857623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85, scope.project.id=434cda3cc623466c9163ef469e5b2e7d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:52.857933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:52.859893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:52.865452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 223/444] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 611 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:52.866250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:52.939015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:52.947271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaffb42-3981-4882-bf2e-ba4d015c7e0c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.980904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88522b53-1a40-4e32-b0cf-33c9d6aaf52a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:52.992172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:52.992827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:52.993106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:52.993374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.006724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:53.012307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:53.013287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:53.017021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 222/445] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:53.019756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.019877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.026153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.026864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.027233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.027575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.028364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:53.028735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.030748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.044374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 224/446] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:52 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:53.046097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.130522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f922d8e-5f24-45eb-88cc-3b9485a9602b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.165957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9776a0a5-b593-4bf4-9161-ece64cee081e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.181449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.182266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.182741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.183206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.184500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:53.185048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.188269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.200243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 223/447] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:53.205994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.206239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.207316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.207316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.207477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.227745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.276033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.284111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.285572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.319453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caa3297-97c3-4a3b-9113-2108865a983b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.352550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.353174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.353454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.353726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.383357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 225/448] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:40:53.389374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.421962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.423340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.440375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.480747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.488036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c263dae8-6ecc-48e4-b51f-d35de9012972 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.521291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.521905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.522181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.522450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] PATH_INFO: `/v3/projects/434cda3cc623466c9163ef469e5b2e7d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.523284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 224/449] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:40:53.523557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] RBAC: Authorizing `identity:get_project(project_id=434cda3cc623466c9163ef469e5b2e7d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.525042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.531410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 226/450] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/projects/434cda3cc623466c9163ef469e5b2e7d => generated 302 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:53.531537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.532303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.626769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9695f0c-bf01-4513-bcdf-7de9baf7e6d1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.630088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fbe9b1-b062-433a-b6a1-90479afca1f5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.643788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.662036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.662653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.662924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.663184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] PATH_INFO: `/v3/projects/efec9f6da9c342038dd7466c677c23ad` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.664286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] RBAC: Authorizing `identity:get_project(project_id=efec9f6da9c342038dd7466c677c23ad)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.664606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.665224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.665502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.665686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.665787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] PATH_INFO: `/v3/groups/5241104f2c73496583e0e0acebfe3d01` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.666865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] RBAC: Authorizing `identity:get_group(group_id=5241104f2c73496583e0e0acebfe3d01)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.667271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 225/451] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/projects/efec9f6da9c342038dd7466c677c23ad => generated 344 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:53.668459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.670001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 227/452] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/groups/5241104f2c73496583e0e0acebfe3d01 => generated 226 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:53.673019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.675558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.704449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.771950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f35de7-5c67-497d-bf6e-7389ccfad236 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.774567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0860f4ad-561a-4d4d-91cb-cedf9b13af4b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.807006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.807553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.807825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.808120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] PATH_INFO: `/v3/groups/ee3deb356179452f8c2600b902d262d3` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.809593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] RBAC: Authorizing `identity:get_group(group_id=ee3deb356179452f8c2600b902d262d3)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.810653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.812856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 226/453] 10.222.0.42 () {56 vars in 1180 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/groups/ee3deb356179452f8c2600b902d262d3 => generated 229 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:53.815275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f8280-a78b-4eaa-b56a-e2f88c0798ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.815871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f8280-a78b-4eaa-b56a-e2f88c0798ff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.816187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f8280-a78b-4eaa-b56a-e2f88c0798ff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.816456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f8280-a78b-4eaa-b56a-e2f88c0798ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.817181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f8280-a78b-4eaa-b56a-e2f88c0798ff None admin] RBAC: Adding query filter params (group.id=5241104f2c73496583e0e0acebfe3d01, role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=434cda3cc623466c9163ef469e5b2e7d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:53.817487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f8280-a78b-4eaa-b56a-e2f88c0798ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.818581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:53.818892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89f8280-a78b-4eaa-b56a-e2f88c0798ff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.824777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 228/454] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/role_assignments?group.id=5241104f2c73496583e0e0acebfe3d01&scope.project.id=434cda3cc623466c9163ef469e5b2e7d&role.id=9d68999390534c65b1cfc53d421f6412 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:53.874921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:53.925529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64107215-bf06-49d2-bbbb-8d285cb15447 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:53.961862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cefc77-6331-413c-bf28-20cdfd864e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:53.962465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cefc77-6331-413c-bf28-20cdfd864e16 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:53.962739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cefc77-6331-413c-bf28-20cdfd864e16 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:53.963003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cefc77-6331-413c-bf28-20cdfd864e16 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:53.963727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cefc77-6331-413c-bf28-20cdfd864e16 None admin] RBAC: Adding query filter params (group.id=ee3deb356179452f8c2600b902d262d3, role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=efec9f6da9c342038dd7466c677c23ad) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:53.964061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cefc77-6331-413c-bf28-20cdfd864e16 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:53.965509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cefc77-6331-413c-bf28-20cdfd864e16 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:53.972515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 227/455] 10.222.0.42 () {56 vars in 1376 bytes} [Fri Sep 16 01:40:53 2022] GET /identity/v3/role_assignments?group.id=ee3deb356179452f8c2600b902d262d3&scope.project.id=efec9f6da9c342038dd7466c677c23ad&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 611 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:56.517206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f993f1c-3a30-4af8-88ed-e64b066830b5 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:56.518044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f993f1c-3a30-4af8-88ed-e64b066830b5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:56.518495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f993f1c-3a30-4af8-88ed-e64b066830b5 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:56.520232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 229/456] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:56.525922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2141ef-505f-4c0a-b95c-5e969f7a706d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:56.526766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2141ef-505f-4c0a-b95c-5e969f7a706d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:56.527230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2141ef-505f-4c0a-b95c-5e969f7a706d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:56.552901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-df2141ef-505f-4c0a-b95c-5e969f7a706d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:56.554394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-df2141ef-505f-4c0a-b95c-5e969f7a706d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:56.646524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 228/457] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:56.759799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d009fe9-dba2-4066-9adb-2889cc171561 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:56.760135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d009fe9-dba2-4066-9adb-2889cc171561 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:56.760378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d009fe9-dba2-4066-9adb-2889cc171561 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:56.762204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 230/458] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:56.767673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaef7f1f-8918-4961-aefc-3b079b5510d5 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:56.768215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaef7f1f-8918-4961-aefc-3b079b5510d5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:56.768679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaef7f1f-8918-4961-aefc-3b079b5510d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:56.797109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-eaef7f1f-8918-4961-aefc-3b079b5510d5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:56.798625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-eaef7f1f-8918-4961-aefc-3b079b5510d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:56.857946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:56.889860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 229/459] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:57.006341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d009fe9-dba2-4066-9adb-2889cc171561 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:57.133765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:57.145157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d009fe9-dba2-4066-9adb-2889cc171561 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:57.197176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9e8cef-a302-47ee-b228-6e471c67d0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:57.197819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9e8cef-a302-47ee-b228-6e471c67d0ae None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.198136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9e8cef-a302-47ee-b228-6e471c67d0ae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.198410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9e8cef-a302-47ee-b228-6e471c67d0ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.215510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9e8cef-a302-47ee-b228-6e471c67d0ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:57.216977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9e8cef-a302-47ee-b228-6e471c67d0ae None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ca9e8cef-a302-47ee-b228-6e471c67d0ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:40:57.238276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:57.239430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:40:57.240552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:40:57.240552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:57.240552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:57.240552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 231/460] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:57.248168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaef7f1f-8918-4961-aefc-3b079b5510d5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:57.301262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6b0486-b0e3-4765-b413-d92037b109fb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.301600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6b0486-b0e3-4765-b413-d92037b109fb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.301802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f6b0486-b0e3-4765-b413-d92037b109fb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.303309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 232/461] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:57.308657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8471a96-681e-49f9-b772-a11eb6562670 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.308941 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8471a96-681e-49f9-b772-a11eb6562670 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.309211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8471a96-681e-49f9-b772-a11eb6562670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.339495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a8471a96-681e-49f9-b772-a11eb6562670 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:57.340721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a8471a96-681e-49f9-b772-a11eb6562670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:57.436063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 233/462] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:57.436063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8471a96-681e-49f9-b772-a11eb6562670 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:57.436856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaef7f1f-8918-4961-aefc-3b079b5510d5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:57.473730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f4a905-6aa0-493a-87fb-4051eb43c7ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:57.474761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4a905-6aa0-493a-87fb-4051eb43c7ca None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.475779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4a905-6aa0-493a-87fb-4051eb43c7ca None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.476361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f4a905-6aa0-493a-87fb-4051eb43c7ca None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.478294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4a905-6aa0-493a-87fb-4051eb43c7ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:57.479348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4a905-6aa0-493a-87fb-4051eb43c7ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:57.481062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f4a905-6aa0-493a-87fb-4051eb43c7ca None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:57.493607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 230/463] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:40:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:57.495315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0126b-c7c9-4142-9de2-953b49ae0efa None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.495754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0126b-c7c9-4142-9de2-953b49ae0efa None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.496190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0126b-c7c9-4142-9de2-953b49ae0efa None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.502840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 231/464] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:57 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:40:57.503814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd0126b-c7c9-4142-9de2-953b49ae0efa None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:57.536789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:57.545956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:57.566061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8471a96-681e-49f9-b772-a11eb6562670 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:57.610806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d624369b-02e5-4a7c-a887-c5d7d7a1a7fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:57.611704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d624369b-02e5-4a7c-a887-c5d7d7a1a7fd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.612188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d624369b-02e5-4a7c-a887-c5d7d7a1a7fd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.612590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d624369b-02e5-4a7c-a887-c5d7d7a1a7fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.623994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d624369b-02e5-4a7c-a887-c5d7d7a1a7fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:57.625181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd0126b-c7c9-4142-9de2-953b49ae0efa None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:57.625986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d624369b-02e5-4a7c-a887-c5d7d7a1a7fd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d624369b-02e5-4a7c-a887-c5d7d7a1a7fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:40:57.643402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:57.644389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:40:57.645218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:40:57.645218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:57.645218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:57.647134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 234/465] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:57.648626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ea0414-9b32-4bce-9ab4-3a84c6e265ad None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.649073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ea0414-9b32-4bce-9ab4-3a84c6e265ad None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.649488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ea0414-9b32-4bce-9ab4-3a84c6e265ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.670309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-89ea0414-9b32-4bce-9ab4-3a84c6e265ad None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:57.671866 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:57.672747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.673094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.673366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-89ea0414-9b32-4bce-9ab4-3a84c6e265ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:57.673467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.674124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:57.675629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:57.702436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 232/466] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:57 2022] POST /identity/v3/users => generated 246 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:57.703753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:57.760145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 235/467] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:40:57.761820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ea0414-9b32-4bce-9ab4-3a84c6e265ad None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:57.828626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:57.840152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8df982-dba1-46fb-82fa-9da416a12d63 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:57.882837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:57.883618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.884107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.884507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:57.885164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:57.885469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:57.887303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:57.893792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:57.903502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 233/468] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:40:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:57.910002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:57.953258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ea0414-9b32-4bce-9ab4-3a84c6e265ad None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:57.991488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:57.992141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:57.992506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:57.992774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.006736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.008286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:40:58.026717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:58.027667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:40:58.028534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:40:58.028534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:58.028534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:58.028534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 236/469] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:58.036484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:58.050103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce084f26-3722-4426-884b-d9f57d953512 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:58.095106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da41fd06-2f9f-4ad0-a2cd-54b7ea2ae4da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:58.096069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41fd06-2f9f-4ad0-a2cd-54b7ea2ae4da None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.097005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41fd06-2f9f-4ad0-a2cd-54b7ea2ae4da None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.097482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41fd06-2f9f-4ad0-a2cd-54b7ea2ae4da None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.098333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da41fd06-2f9f-4ad0-a2cd-54b7ea2ae4da None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.100353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da41fd06-2f9f-4ad0-a2cd-54b7ea2ae4da None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.129201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 234/470] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:57 2022] POST /identity/v3/users => generated 248 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:58.138217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460d974-e904-4622-868a-c8b441ae22bd None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.139249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460d974-e904-4622-868a-c8b441ae22bd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.139887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9460d974-e904-4622-868a-c8b441ae22bd None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.141528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 235/471] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:40:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:40:58.148733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51172d4-7f88-4220-a63e-1a286d9e8dd1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.149027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51172d4-7f88-4220-a63e-1a286d9e8dd1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.149271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51172d4-7f88-4220-a63e-1a286d9e8dd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.159826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4e72a8-6315-4631-85f5-d2a6a6157515 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:58.173341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:58.174618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e51172d4-7f88-4220-a63e-1a286d9e8dd1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:40:58.175921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e51172d4-7f88-4220-a63e-1a286d9e8dd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:40:58.178701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:58.204907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:58.205826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.206243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.206704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.207507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:58.207987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.209567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.230546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 237/472] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:40:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:40:58.235969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:58.262870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 236/473] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:40:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:40:58.263794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51172d4-7f88-4220-a63e-1a286d9e8dd1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:58.376827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93e576-0bd9-4408-ab09-a7a4be4ee5e1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:58.428252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:58.429514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.429986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.430867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.438202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51172d4-7f88-4220-a63e-1a286d9e8dd1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:58.446571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.448873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:40:58.471480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:58.472504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:40:58.473341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:40:58.473341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:40:58.473341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:40:58.473506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 238/474] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:40:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:58.479729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:58.490536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:58.491497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.492014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.492475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.493442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.495126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.506844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:58.510112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:58.526398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 237/475] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:58 2022] POST /identity/v3/users => generated 248 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:58.596716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:58.644558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa80a1b4-799d-447f-8f97-b4ef6865e5d4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:58.702977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:58.703599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.703874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.704166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.704829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:58.705129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.706526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.733209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 239/476] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:40:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:40:58.739120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:58.765076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038faa8e-d1a2-487a-9ced-faabf05587f2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:58.823903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:58.824637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.825022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.825308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.825995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.827667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.836874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.sql.core [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 16 01:40:58.836874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 16 01:40:58.836874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [parameters: {'id': 'a17afb5558924151a64e6457276924bb', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 16 01:40:58.836874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81424) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 16 01:40:58.849187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application conn.query(q) Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result.read() Sep 16 01:40:58.855965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return ref.to_dict() Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 01:40:58.857559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application next(self.gen) Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application yield resource Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application next(self.gen) Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.session.rollback() Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.force_reraise() Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise self.value Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application session.commit() Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 16 01:40:58.858816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._prepare_impl() Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.session.flush() Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._flush(objects) Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application compat.raise_( Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application flush_context.execute() Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rec.execute(self) Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 16 01:40:58.860071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 01:40:58.860901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application conn.query(q) Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result.read() Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 01:40:58.861682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [parameters: {'id': 'a17afb5558924151a64e6457276924bb', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:58.862445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 16 01:40:58.865999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:58.867305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 238/477] 10.222.0.42 () {60 vars in 1131 bytes} [Fri Sep 16 01:40:58 2022] POST /identity/v3/roles => generated 144 bytes in 261 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:40:58.867305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:58.872543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7faea4-56c2-4756-bc0a-dca323f82814 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:58.919003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:58.920051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:58.920339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:58.920635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:58.922240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:58.923449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:58.946307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:58.960019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 240/478] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:40:58 2022] POST /identity/v3/users => generated 249 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:40:59.007227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6b1412-d5a0-4be2-a441-e2b511c84056 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:59.048039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c398a05-cf39-4478-89b5-336b13cd4f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:59.048683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c398a05-cf39-4478-89b5-336b13cd4f61 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:59.048961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c398a05-cf39-4478-89b5-336b13cd4f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:59.049230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c398a05-cf39-4478-89b5-336b13cd4f61 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:59.062482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c398a05-cf39-4478-89b5-336b13cd4f61 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-5c398a05-cf39-4478-89b5-336b13cd4f61 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:40:59.068160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:40:59.069053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:40:59.069961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 239/479] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:40:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:40:59.075628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:40:59.185522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4af4e86-6cd9-43aa-b801-56f241751e5f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:40:59.226295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5ee61a-1b01-48a5-93a5-e3ebc7da55e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:40:59.226907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ee61a-1b01-48a5-93a5-e3ebc7da55e7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:40:59.227184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ee61a-1b01-48a5-93a5-e3ebc7da55e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:40:59.227447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5ee61a-1b01-48a5-93a5-e3ebc7da55e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:40:59.228130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5ee61a-1b01-48a5-93a5-e3ebc7da55e7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:40:59.228428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5ee61a-1b01-48a5-93a5-e3ebc7da55e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:40:59.229909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5ee61a-1b01-48a5-93a5-e3ebc7da55e7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:40:59.236541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:40:59.239894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 241/480] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:40:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:00.497948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed480a3-3afb-42e0-976d-c80d560c70e4 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:00.498647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed480a3-3afb-42e0-976d-c80d560c70e4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:00.498647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed480a3-3afb-42e0-976d-c80d560c70e4 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:00.500402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 240/481] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:00.509775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf80e0f-e6fd-432e-91a2-913dd592b634 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:00.510322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf80e0f-e6fd-432e-91a2-913dd592b634 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:00.510322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf80e0f-e6fd-432e-91a2-913dd592b634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:00.532975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5cf80e0f-e6fd-432e-91a2-913dd592b634 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:00.534273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5cf80e0f-e6fd-432e-91a2-913dd592b634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:00.633891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 242/482] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:01.038579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed480a3-3afb-42e0-976d-c80d560c70e4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:01.058781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63ac7d9-165b-4239-9e3f-df115b0b0d00 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.059396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63ac7d9-165b-4239-9e3f-df115b0b0d00 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.059879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63ac7d9-165b-4239-9e3f-df115b0b0d00 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.062399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 243/483] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:01.067482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a69da-4535-4ff5-9bef-a137d3d673da None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.068025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a69da-4535-4ff5-9bef-a137d3d673da None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.068494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a69da-4535-4ff5-9bef-a137d3d673da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.098156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0b2a69da-4535-4ff5-9bef-a137d3d673da None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:01.099642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0b2a69da-4535-4ff5-9bef-a137d3d673da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:01.204749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 244/484] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:41:01.210665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed480a3-3afb-42e0-976d-c80d560c70e4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:01.248310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:01.260633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa9edb7-ec44-4275-9022-dbbacf354499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:01.261851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9edb7-ec44-4275-9022-dbbacf354499 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.262549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9edb7-ec44-4275-9022-dbbacf354499 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.262818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa9edb7-ec44-4275-9022-dbbacf354499 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.274388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa9edb7-ec44-4275-9022-dbbacf354499 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-0fa9edb7-ec44-4275-9022-dbbacf354499 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:01.279077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:01.280528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:01.281737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 241/485] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:01.287126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a69da-4535-4ff5-9bef-a137d3d673da None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:01.370257 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:01.429827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a69da-4535-4ff5-9bef-a137d3d673da None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:01.458567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abac8b3-a80f-415c-bd0b-02c0182879a9 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.458896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abac8b3-a80f-415c-bd0b-02c0182879a9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.459099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abac8b3-a80f-415c-bd0b-02c0182879a9 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.460451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 242/486] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:01.468333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec56c097-9127-489b-8b92-c8cc802bd2ee None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.468569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec56c097-9127-489b-8b92-c8cc802bd2ee None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.468779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec56c097-9127-489b-8b92-c8cc802bd2ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.484321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:01.484936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.485212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.485474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.486135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:01.486432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:01.487788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:01.498327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-ec56c097-9127-489b-8b92-c8cc802bd2ee None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:01.500356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ec56c097-9127-489b-8b92-c8cc802bd2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:01.507776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 245/487] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:01.508835 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:01.612020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 243/488] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:01.662830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9001b280-881f-4432-a38c-e295147f8250 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:01.686753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:01.698974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:01.699892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.700419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.700881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.712444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:01.714230 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:01.729667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:01.730624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:01.731474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:01.731474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:01.731474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:01.734184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 246/489] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:01.738983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec56c097-9127-489b-8b92-c8cc802bd2ee None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:01.750383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:01.914390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec56c097-9127-489b-8b92-c8cc802bd2ee None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:01.925196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:01.958230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cac0b52-8f13-4571-9967-4ddbf1d751e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:01.958230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cac0b52-8f13-4571-9967-4ddbf1d751e6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.958230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cac0b52-8f13-4571-9967-4ddbf1d751e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.958230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cac0b52-8f13-4571-9967-4ddbf1d751e6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.958635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699dd14b-7e56-4192-baa3-45c844cc9ea9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:01.969727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cac0b52-8f13-4571-9967-4ddbf1d751e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-1cac0b52-8f13-4571-9967-4ddbf1d751e6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:01.975184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:01.976311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:01.977086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:01.981550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 244/490] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:01.982399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb2578-a4e4-48fa-ad53-63fb6dab7a6b None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:01.982688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb2578-a4e4-48fa-ad53-63fb6dab7a6b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:01.983009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb2578-a4e4-48fa-ad53-63fb6dab7a6b None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:01.984564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 245/491] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:01.985122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb2578-a4e4-48fa-ad53-63fb6dab7a6b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:01.998549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.000085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.000539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.001132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.002356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:02.003102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.004828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.018900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 247/492] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:02.019732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.053648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.091651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb2578-a4e4-48fa-ad53-63fb6dab7a6b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.117457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4c34e7-bfaa-4ef6-9b51-38a1e144967a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.125527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816b26d3-d966-4e0b-bd74-faa4d01fceba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.126141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816b26d3-d966-4e0b-bd74-faa4d01fceba None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.126414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816b26d3-d966-4e0b-bd74-faa4d01fceba None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.126675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816b26d3-d966-4e0b-bd74-faa4d01fceba None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.136964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816b26d3-d966-4e0b-bd74-faa4d01fceba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-816b26d3-d966-4e0b-bd74-faa4d01fceba None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:02.142499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:02.143468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:02.144617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 246/493] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:02.145286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdfb0e8-2b6b-44b6-9507-59fc2a178d90 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.145286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdfb0e8-2b6b-44b6-9507-59fc2a178d90 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.145286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fdfb0e8-2b6b-44b6-9507-59fc2a178d90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.153236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46dd5e2c-a23f-4084-810a-5446be1cdc2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.153829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dd5e2c-a23f-4084-810a-5446be1cdc2f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.154100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dd5e2c-a23f-4084-810a-5446be1cdc2f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.154362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46dd5e2c-a23f-4084-810a-5446be1cdc2f None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.155122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dd5e2c-a23f-4084-810a-5446be1cdc2f None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:02.155419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dd5e2c-a23f-4084-810a-5446be1cdc2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.156797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46dd5e2c-a23f-4084-810a-5446be1cdc2f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.162861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1fdfb0e8-2b6b-44b6-9507-59fc2a178d90 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:02.164094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1fdfb0e8-2b6b-44b6-9507-59fc2a178d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:02.164834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 248/494] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:02.166001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bae24e-c8c3-4f1e-8bdb-82327272adfb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.166283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bae24e-c8c3-4f1e-8bdb-82327272adfb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.166549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bae24e-c8c3-4f1e-8bdb-82327272adfb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.167873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 249/495] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:02.170121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bae24e-c8c3-4f1e-8bdb-82327272adfb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.188685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.231553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 247/496] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:02.232408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdfb0e8-2b6b-44b6-9507-59fc2a178d90 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.267457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.275100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bae24e-c8c3-4f1e-8bdb-82327272adfb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.310530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.311185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.311598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.312081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.319860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.321324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:02.335003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:02.335918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:02.336843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 250/497] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:02.341083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fdfb0e8-2b6b-44b6-9507-59fc2a178d90 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.341699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.377859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb99a685-7f82-4685-adf3-c1bb673c9797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.378462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99a685-7f82-4685-adf3-c1bb673c9797 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.378736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99a685-7f82-4685-adf3-c1bb673c9797 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.379001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb99a685-7f82-4685-adf3-c1bb673c9797 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.379677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99a685-7f82-4685-adf3-c1bb673c9797 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:02.379967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99a685-7f82-4685-adf3-c1bb673c9797 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.381330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb99a685-7f82-4685-adf3-c1bb673c9797 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.395988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 248/498] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:02.395988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9328d7b2-6be5-4fbe-95ea-1279dac04c58 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.395988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9328d7b2-6be5-4fbe-95ea-1279dac04c58 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.395988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9328d7b2-6be5-4fbe-95ea-1279dac04c58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.412991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-9328d7b2-6be5-4fbe-95ea-1279dac04c58 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:02.414176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9328d7b2-6be5-4fbe-95ea-1279dac04c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:02.430024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.452088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f8af53-84ca-4712-a97f-88f11cc95141 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.487620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 249/499] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:02.488404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9328d7b2-6be5-4fbe-95ea-1279dac04c58 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.500524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.501708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.502203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.502906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.516653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.518105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:02.532736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:02.533623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:02.534442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:02.534442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:02.534442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:02.534442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 251/500] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:02.534661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.546334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.601037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9328d7b2-6be5-4fbe-95ea-1279dac04c58 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.635898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.636773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.637048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.637313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.638025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:02.638322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.638494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b1ecad-16e1-4a92-862c-ccc60500e35b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.639840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.673397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.674053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.674277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.674599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.676558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 250/501] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:02.681992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.685760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.688130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.693744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:02.702990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:02.708810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:02.709905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:02.709905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:02.709905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:02.709905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 252/502] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:02.709905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.768036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.797610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cfcc14-70ed-4cf1-bf1a-dc5276f75ee9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.808277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e092c62-c8d0-446a-a8f3-9f240cc71ff7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.832721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.833343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.833621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.833890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.842938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.843569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.843845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.843956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:02.844134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.844758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:02.845053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.846403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:02.848715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:02.849729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:02.850728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 251/503] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:02.851498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.863386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 253/504] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:02.864219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:02.918734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.952489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46875fe8-eed5-4569-a1a8-936f998d40ff None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.961614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c54201-f011-4caa-afb1-c4e510e179a7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:02.981058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:02.988135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.988735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.989013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.989283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.995863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:02.996477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:02.996750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:02.997019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:02.997641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:02.997933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:02.999983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.010879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 254/505] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:03.012239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.056742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.058205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.109335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb9656a-ee2d-49cd-a70f-bfaf32cef23d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.122363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.142795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.143407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.143678 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.143962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.155808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.157399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 252/506] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:03.157507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.158234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:03.172225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:03.173154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:03.173978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:03.173978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:03.173978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:03.173978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 255/507] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:03.174885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.207690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.258945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444cc268-0fc6-4146-8c31-84759940fef1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.276840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f6bcfb-2828-41d8-80f3-f23b914467f9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.296409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.297438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.297961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.298464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.299610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:03.299929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.301715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.309782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 253/508] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:03.310472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.314838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.315464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.315733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.316019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] PATH_INFO: `/v3/users/glance` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.323517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.324957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:03.336622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:03.337524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:03.338547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 256/509] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/users/glance => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:03.339599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.347067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.416271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176d9995-56ee-4f76-80b6-db50a4a29487 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.427548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.437214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac08fd-69b3-4615-9fa0-e4ab8f38acfd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.449398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.450016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.450298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.450568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.458214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.459593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.470197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.470813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.471090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.471358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:03.472489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:03.473400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:03.474259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 254/510] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/users/cinder => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:03.474795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.487213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.488686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:03.502584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:03.503460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:03.504336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:03.504336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:03.504336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:03.504336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 257/511] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:03.504553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.571271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.581530 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627bb6ce-f867-4d43-95d1-721f48a261d6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.604909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961283dd-65d4-4c16-86d7-8b61a5312c22 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.614665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.615271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.615550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.615823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.616493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:03.616793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.618168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.630512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 255/512] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:03.631318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.641966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.642593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.642873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.643505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.654030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.655494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.657968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:03.673198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:03.674727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:03.675759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:03.675759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:03.675759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:03.680110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 258/513] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:03.681282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.732627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03400d4-a5b1-4c70-b6ae-a7af416aafe7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.766663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.767261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.767540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.767808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.768554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:03.768861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.770267 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.785126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0393d7c6-cb4f-4dbf-adec-6d338506e366 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.787323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 256/514] 10.222.0.42 () {56 vars in 1137 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:03.788069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.802308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.819499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.820197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.820539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.820868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.821651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:03.821952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.823362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.861305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 259/515] 10.222.0.42 () {56 vars in 1137 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:03.862319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.896012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7b0067-e4d7-406c-b4c4-861989287afb None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:03.917447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:03.930161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:03.930770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:03.931047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:03.931318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:03.931976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:03.932285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:03.933646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:03.945063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 257/516] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:03.945854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:03.967058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68015c-e823-467f-b2a9-cba8d872c8de None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.001841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.002358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.002634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.002902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.003544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.005016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.010859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.025476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 260/517] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:41:03 2022] POST /identity/v3/users => generated 247 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:04.026379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.051310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63047325-a421-4dde-b496-9b8458d70c6f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.090593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.091212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.091495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.091774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.092499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:04.092942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.094344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.112174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 258/518] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:04.112174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.156582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4de046-a646-4e33-b93f-1a38687f17a1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.167866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.195987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.196595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.196867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.197126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.229327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868ca0ee-36f0-4ab8-9859-787fd233ece9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.259273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.268693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.269307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.269589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.269864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.284716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.286198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.351959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.354491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.392656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 261/519] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:04.392656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.403489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.464370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 259/520] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:04.465231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.494571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.511277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b635ab-a384-4b8e-a7c6-61f9692d740f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.549391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.550069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.550323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.550591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.557455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.558908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.568301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebc1446-951a-4dd8-91ce-50a2293394ad None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:04.571223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:04.572279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 262/521] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:04.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.609055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.609781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.610061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.610342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.618448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.620037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:04.632813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:04.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:04.634654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 260/522] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/users/neutron => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:04.635211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.654509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.687191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551c0181-1f06-4567-9e8f-2c50ecc11ed5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.723076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.723815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.724128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.724426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.730172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.737700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.739150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.743374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf489e3-3b9a-4454-83da-f78e9cf7389f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:04.754516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:04.755633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:04.761626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:04.761626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:04.761626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:04.761626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 263/523] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:04.761626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.786406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.787131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.787478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.787833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.802022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.803569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:04.823546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:04.824905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:04.825729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:04.825729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:04.825729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:04.825729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 261/524] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:04.825729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.874900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4dd27fb-7f8c-4677-aee8-e4eceb86a214 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.901841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.914522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.915258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.915637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.916184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.917070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:04.917475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.918995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.931525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22945a8b-d9d8-434f-80b4-1e2f959ad571 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:04.944548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 264/525] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:04.944548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:04.974638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:04.975851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:04.976273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:04.976547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:04.977275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:04.977580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:04.979001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:04.988473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:04.999395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 262/526] 10.222.0.42 () {56 vars in 1139 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:05.000243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.052664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c63765e-bfde-43e2-a222-42915f07316a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.089841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.090665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.090950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.091222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.092025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:05.092844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.094332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.108756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 265/527] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:05.109579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.111655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5554f1b-4c6d-484c-a967-bec2187e031c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.149450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.150386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.150903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.151363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.152151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:05.153151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:05.153560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.156137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.167980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 263/528] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:05.168341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.217027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81edcb77-dc96-4b74-a0bf-0e5986917cb9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.228454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:05.252688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.253492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.253599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.253821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.254543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=603c81d781754201a5ac5b7d9b84fab6) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:05.254837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.256296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.269155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 266/529] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/role_assignments?user.id=603c81d781754201a5ac5b7d9b84fab6&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:05.269155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.286154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2422a739-e8b2-47c3-b862-bcbccb152cca None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.334192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.335207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.335815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.336284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.414154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b6ae3b-9765-47b5-80cb-03f29e339021 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.425066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:05.441963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.444325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.461723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.462352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.462636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.462905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.477965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.480192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:05.496154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:05.497015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:05.497850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 267/530] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:05.498299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.518564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:05.571320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 264/531] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:05.571320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.618075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6b4330-337f-4c9f-91d7-86131c7781a0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.656799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.657715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.658138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.658687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.666356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.668003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:05.680861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:05.681893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:05.682946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:05.683447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59dbe2d-a225-4846-ab8d-9bd5c67f0e4e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.685320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 268/532] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:05.687198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.722575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:05.733162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.734139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.734616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.735032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.735902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:05.736370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.738043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.766860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 265/533] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:05.768527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.791693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:05.819677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47952c4e-2c62-45f6-a54e-41bd371c65b7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.862748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.863560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.863924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.864780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.878568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.880697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:05.900289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:05.901239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:05.902105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:05.902105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:05.902105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:05.902105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 269/534] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:05.907405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:05.907542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a238b64-51aa-498d-8ab4-5a8e3434de00 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:05.947075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:05.947814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:05.948249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:05.948524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:05.949211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:05.949511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:05.950988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:05.963304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 266/535] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:05.968218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:06.015251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:06.039667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bc0313-5a15-474f-a709-bd368750f426 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:06.071564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:06.080285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:06.081270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.081691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.082103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.082980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=30b10ebdc87a4747bef7fc60b3cec9c8) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:06.083425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:06.085109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:06.091333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 270/536] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/role_assignments?user.id=30b10ebdc87a4747bef7fc60b3cec9c8&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:06.092856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:06.098133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283b090-8517-4a85-86c4-fa8b76ab5e6c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:06.135896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:06.136863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.137213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.137553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.138297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:06.138668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:06.140832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:06.160769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 267/537] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:06.162271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:06.215929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a8c71-cc92-4541-be87-e4f90a453785 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:06.256590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29f1bf0-264f-46f5-beb1-8f6371b0f1f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:06.257606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29f1bf0-264f-46f5-beb1-8f6371b0f1f1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.258083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29f1bf0-264f-46f5-beb1-8f6371b0f1f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.258819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29f1bf0-264f-46f5-beb1-8f6371b0f1f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.259763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29f1bf0-264f-46f5-beb1-8f6371b0f1f1 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=d7b63222dc2846f5a42208877b0de123) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:06.260104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29f1bf0-264f-46f5-beb1-8f6371b0f1f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:06.261861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29f1bf0-264f-46f5-beb1-8f6371b0f1f1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:06.268482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 271/538] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:06 2022] GET /identity/v3/role_assignments?user.id=d7b63222dc2846f5a42208877b0de123&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:06.269837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ebca74-8778-4b04-8def-8348e8c0413d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.270119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ebca74-8778-4b04-8def-8348e8c0413d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.270391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ebca74-8778-4b04-8def-8348e8c0413d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.272085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 272/539] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:06.279102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e9fa8-8fed-47ba-8873-d75f28676cd5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.280314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e9fa8-8fed-47ba-8873-d75f28676cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.280818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057e9fa8-8fed-47ba-8873-d75f28676cd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.295614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1db926-5cc8-4f32-99c3-cb16f9f2b73e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:06.302209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:06.319330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-057e9fa8-8fed-47ba-8873-d75f28676cd5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:06.322943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-057e9fa8-8fed-47ba-8873-d75f28676cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:06.387168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:06.387873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.388234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.388501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.405759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:06.407586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:06.413932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:06.428242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 273/540] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:06.433429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:06.434888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:06.436418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 268/541] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:06.442168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057e9fa8-8fed-47ba-8873-d75f28676cd5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:06.575448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057e9fa8-8fed-47ba-8873-d75f28676cd5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:06.612112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:06.612720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.612997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.613264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.614024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:06.614326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:06.615809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:06.618840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:06.635214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 274/542] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:06.637313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:06.810667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6ae56a-7e43-492a-bc63-33bdfac65ec8 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:06.833125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:06.863604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:06.864245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:06.864525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:06.864799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:06.865524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=6a343a260fa2415e84caf959933a6c13) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:06.865834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:06.867425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:06.885656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 269/543] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:06 2022] GET /identity/v3/role_assignments?user.id=6a343a260fa2415e84caf959933a6c13&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 259 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:06.983794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d70fb4b-240d-4d40-9abb-6d276f658652 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:07.023766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:07.024432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:07.024757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:07.025036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:07.040968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:07.047108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:07.048099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:07.048886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 275/544] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:07.056209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:07.154157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:07.199245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046159b1-42cd-4f6b-92a6-0f297ae6addc None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:07.235527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:07.236160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:07.236443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:07.236714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:07.237401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:07.237760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:07.239168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:07.254265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 270/545] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:07.260058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:07.360933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:07.423520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68809efd-159a-4a3c-b7ae-9835b778261c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:07.470122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:07.470748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:07.471022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:07.471284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:07.484408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:07.485998 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:07.505340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:07.506284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:07.507122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:07.507122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:07.507122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:07.507122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 276/546] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:07.512503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:07.627183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109507c-e846-41c7-bd36-c4991fdca1f2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:07.665297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:07.678166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d96a5b-1cc0-45a0-9eab-bfe6a9a237bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:07.678776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d96a5b-1cc0-45a0-9eab-bfe6a9a237bc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:07.679056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d96a5b-1cc0-45a0-9eab-bfe6a9a237bc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:07.679324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d96a5b-1cc0-45a0-9eab-bfe6a9a237bc None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:07.680064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d96a5b-1cc0-45a0-9eab-bfe6a9a237bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:07.680369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d96a5b-1cc0-45a0-9eab-bfe6a9a237bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:07.681726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d96a5b-1cc0-45a0-9eab-bfe6a9a237bc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:07.698518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 271/547] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:07.699979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:07.810720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b472ce-5039-4d92-803e-690c2993916b None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:07.811738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b472ce-5039-4d92-803e-690c2993916b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:07.812166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b472ce-5039-4d92-803e-690c2993916b None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:07.814418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 272/548] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:07.820267 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a1bf61-1ebe-4fd6-9fda-eb57afe845e8 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:07.820547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a1bf61-1ebe-4fd6-9fda-eb57afe845e8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:07.820815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a1bf61-1ebe-4fd6-9fda-eb57afe845e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:07.830370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53423757-e8a3-44a4-9b42-b2be0673f87e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:07.841216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:07.844475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c8a1bf61-1ebe-4fd6-9fda-eb57afe845e8 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:07.846629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c8a1bf61-1ebe-4fd6-9fda-eb57afe845e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:07.868978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:07.869608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:07.869886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:07.870160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:07.892365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:07.893894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:07.915603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:07.917249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:07.918518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:07.923998 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 277/549] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:07 2022] GET /identity/v3/users/swift => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:07.925279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:07.957547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 273/550] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:41:08.077472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850fe5e6-188a-4422-bbe9-97e1bd608de2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:08.126271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:08.126941 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:08.127225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:08.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:08.128264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:08.128577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:08.130057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:08.158664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 278/551] 10.222.0.42 () {56 vars in 1135 bytes} [Fri Sep 16 01:41:07 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:08.160106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a1bf61-1ebe-4fd6-9fda-eb57afe845e8 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:08.281182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a1bf61-1ebe-4fd6-9fda-eb57afe845e8 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:08.297621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:08.319809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:08.320765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:08.321235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:08.321690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:08.325335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:08.368930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:08.455711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:08.456105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b83ba2-ed39-406c-b333-a16337ff60b3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:08.457859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:08.491866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:08.492514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:08.492788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:08.493053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:08.504281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:08.509060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:08.510087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:08.510893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 279/552] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:08.521003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:08.634131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 274/553] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 470 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:08.635324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:08.664340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:08.679149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:08.721909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d31ccc-0414-4f3d-a2f6-bba4259399dd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:08.742427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc349db-433c-4aa2-a357-34d906767030 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:08.778965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:08.779919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:08.780236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:08.780557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:08.791140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:08.792611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:08.820882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:08.821240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:08.822513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:08.823362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:08.823362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:08.823362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:08.823523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:08.823523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:08.823523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:08.823707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 275/554] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:08.823772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:08.823772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:08.825595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:08.828957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:08.847547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 280/555] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:08.860932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:08.910549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:08.968213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfa9fc2-9b6c-4f6c-9190-e854969abcfe None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.001500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1b116b-3fb2-4105-9241-dd28f636f6ca None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.012653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f44fd15-26bb-46f8-a7c2-723560e46810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.014524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f44fd15-26bb-46f8-a7c2-723560e46810 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.014813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f44fd15-26bb-46f8-a7c2-723560e46810 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.015085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f44fd15-26bb-46f8-a7c2-723560e46810 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.015761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f44fd15-26bb-46f8-a7c2-723560e46810 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:09.016099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f44fd15-26bb-46f8-a7c2-723560e46810 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.018076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f44fd15-26bb-46f8-a7c2-723560e46810 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.033978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 276/556] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:09.035358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9365ac4-2ed5-4c50-8a29-a5ec8ec87cae None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.035638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9365ac4-2ed5-4c50-8a29-a5ec8ec87cae None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.035905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9365ac4-2ed5-4c50-8a29-a5ec8ec87cae None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.037669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 277/557] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:09.041025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9365ac4-2ed5-4c50-8a29-a5ec8ec87cae None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.063549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b9441d-4e5a-4489-9056-0560bdcefd62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.064258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9441d-4e5a-4489-9056-0560bdcefd62 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.064797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9441d-4e5a-4489-9056-0560bdcefd62 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.065073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b9441d-4e5a-4489-9056-0560bdcefd62 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.081064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b9441d-4e5a-4489-9056-0560bdcefd62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.083142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b9441d-4e5a-4489-9056-0560bdcefd62 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.093595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-74b9441d-4e5a-4489-9056-0560bdcefd62 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:09.104194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:09.105120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:09.105975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:09.105975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:09.105975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:09.111644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 281/558] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:09.113924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fed5aa-fff5-49ed-bb21-cfb1d73429ec None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.114217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fed5aa-fff5-49ed-bb21-cfb1d73429ec None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.114491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fed5aa-fff5-49ed-bb21-cfb1d73429ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.137323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-53fed5aa-fff5-49ed-bb21-cfb1d73429ec None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:09.138583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-53fed5aa-fff5-49ed-bb21-cfb1d73429ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:09.165931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9365ac4-2ed5-4c50-8a29-a5ec8ec87cae None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.192257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:09.202358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28ca3b6-13b8-4c40-8369-ac23d2aa29fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.202992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ca3b6-13b8-4c40-8369-ac23d2aa29fe None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.203270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ca3b6-13b8-4c40-8369-ac23d2aa29fe None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.203540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28ca3b6-13b8-4c40-8369-ac23d2aa29fe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.210303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ca3b6-13b8-4c40-8369-ac23d2aa29fe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.211725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28ca3b6-13b8-4c40-8369-ac23d2aa29fe None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.223963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 282/559] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:09.223963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fed5aa-fff5-49ed-bb21-cfb1d73429ec None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-e28ca3b6-13b8-4c40-8369-ac23d2aa29fe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:09.230549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:09.231512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:09.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:09.236858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 278/560] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:09.238243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4a551-5cc9-4fea-bcf8-bdab48e08eea None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.238597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4a551-5cc9-4fea-bcf8-bdab48e08eea None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.238939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a4a551-5cc9-4fea-bcf8-bdab48e08eea None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.240798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 279/561] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:09.241581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a4a551-5cc9-4fea-bcf8-bdab48e08eea None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.335669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fed5aa-fff5-49ed-bb21-cfb1d73429ec None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.358712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a4a551-5cc9-4fea-bcf8-bdab48e08eea None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.377215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:09.385585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3294b09-69d8-4a13-a1f9-10b41076e870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.386710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3294b09-69d8-4a13-a1f9-10b41076e870 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.387534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3294b09-69d8-4a13-a1f9-10b41076e870 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.388017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3294b09-69d8-4a13-a1f9-10b41076e870 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.389636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3294b09-69d8-4a13-a1f9-10b41076e870 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:09.390143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3294b09-69d8-4a13-a1f9-10b41076e870 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.392749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3294b09-69d8-4a13-a1f9-10b41076e870 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.399656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.400402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.400794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.401162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.401945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:09.402751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.403847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.414078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 283/562] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:09.416217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add6817-c37f-4a59-98a0-8276268c33b1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.416658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add6817-c37f-4a59-98a0-8276268c33b1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.417482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add6817-c37f-4a59-98a0-8276268c33b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.424069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 280/563] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:09.425068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.440311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5add6817-c37f-4a59-98a0-8276268c33b1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:09.441971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5add6817-c37f-4a59-98a0-8276268c33b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:09.476559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:09.540426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 284/564] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:09.540950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6cefa0-406a-4ca0-a8cd-67c8481dd000 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.541922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add6817-c37f-4a59-98a0-8276268c33b1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.577387 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.578011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.578291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.578560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.586479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.587909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:09.601299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:09.602268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:09.603188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:09.607079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 281/565] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:09.608005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.661801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add6817-c37f-4a59-98a0-8276268c33b1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.682791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:09.697624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.698215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.698489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.698753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.699458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=af7a48729497436784b988c6291cfa5d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:09.699756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.701204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.706940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 285/566] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/role_assignments?user.id=af7a48729497436784b988c6291cfa5d&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:09.707537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.725181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18353b40-f1d7-4fbf-9ebf-59d415f10b26 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.737864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:09.777612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccfd6df-71a0-49c4-88a3-888195ab26d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.778245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccfd6df-71a0-49c4-88a3-888195ab26d1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.778519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccfd6df-71a0-49c4-88a3-888195ab26d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.778786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccfd6df-71a0-49c4-88a3-888195ab26d1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.791612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccfd6df-71a0-49c4-88a3-888195ab26d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-fccfd6df-71a0-49c4-88a3-888195ab26d1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:09.797777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:09.798755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:09.799561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 282/567] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:09.800279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a062e-b63c-4bfe-a50d-0879e88ba5ca None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.800854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a062e-b63c-4bfe-a50d-0879e88ba5ca None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.800854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53a062e-b63c-4bfe-a50d-0879e88ba5ca None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.807688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 283/568] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:09 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:09.808659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53a062e-b63c-4bfe-a50d-0879e88ba5ca None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:09.846338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a194743-248b-465d-8dcf-30456eab7801 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.884626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9867c71e-c016-4369-a369-b64e46b3a3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.885289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867c71e-c016-4369-a369-b64e46b3a3b8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.885505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867c71e-c016-4369-a369-b64e46b3a3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.885776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9867c71e-c016-4369-a369-b64e46b3a3b8 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.886776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867c71e-c016-4369-a369-b64e46b3a3b8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:09.887080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867c71e-c016-4369-a369-b64e46b3a3b8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.888545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9867c71e-c016-4369-a369-b64e46b3a3b8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.915966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 286/569] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:09.915966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0b55d2-b117-4980-8c02-9d9577fc575f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.915966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0b55d2-b117-4980-8c02-9d9577fc575f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.915966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0b55d2-b117-4980-8c02-9d9577fc575f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.928824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ee0b55d2-b117-4980-8c02-9d9577fc575f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:09.930018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ee0b55d2-b117-4980-8c02-9d9577fc575f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:09.930701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53a062e-b63c-4bfe-a50d-0879e88ba5ca None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:09.966882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:09.967576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:09.967889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:09.968411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:09.969099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:09.969404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:09.970774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:09.972393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:09.979112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 284/570] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:09.979843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.010685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:10.014248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 287/571] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:10.015402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0b55d2-b117-4980-8c02-9d9577fc575f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.116487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3f692b-52c3-4c1a-99e3-802cfd1dc424 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.124271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0b55d2-b117-4980-8c02-9d9577fc575f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.160440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.161052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.161324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.161588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.163760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.165033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.165511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.165980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.172957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.174409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.180150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:10.187055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:10.188006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:10.188828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:10.189996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:10.190865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:10.191838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:10.191838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:10.191838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:10.193879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 285/572] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:10.194953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.195687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 288/573] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:10.196650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.224059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:10.307133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974e365c-4786-4a89-a944-1ed21cfa2788 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.314386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:10.329091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74c59c1-c112-4109-a783-2e1b7f89d61c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.343760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.344592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.344723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.344921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.345555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:10.345916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.347350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.363434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 289/574] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:10.363434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.369995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.370630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.370907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.371178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.384004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.385336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:10.401021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:10.401979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:10.402848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:10.402848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:10.402848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:10.407089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 286/575] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:10.408053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.460651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:10.478773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22e9779-0da6-4400-966a-4f4726998fef None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.516853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd88a9f-7291-4a94-97ec-189b15322282 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.520111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.520719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.520993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.521257 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.521912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:10.522205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.523554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.534305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 290/576] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:10.535115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.558243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.558873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.559150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.559421 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.568010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:10.571837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:10.577196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:10.578118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:10.578930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 287/577] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:10.579622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.644925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7136a5-ba30-464d-883e-1239aef962c0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.679069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.679687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.680255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.680255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.687729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.689201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.693026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02616513-7a08-4ce2-a769-07a1840c0bac None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:10.701548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:10.702459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:10.703319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 291/578] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:10.704397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.713800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:10.732317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.733546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.733880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.734204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.734998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:10.735353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.736898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.754442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 288/579] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:10.754442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.817584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51f81d3-75de-4e0c-846c-b5fdc7898a24 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.822119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:10.854788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.855411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.855691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.855983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.865787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949cc66c-df2f-40d7-a734-9d2ac826611e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:10.867884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.869570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:10.885428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:10.887213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:10.888360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:10.888360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:10.888360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:10.890867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 292/580] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:10.891791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.903196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:10.903927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:10.904332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:10.905365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:10.906286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:10.906688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:10.908349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:10.921614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 289/581] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:10.921614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:10.958005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.008614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae61c87-1ebc-447d-b070-e824ae9e0cd9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.032645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fc81aa-50cd-457a-be7a-04a98df82775 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.045415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.045901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.046172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.046440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.047122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:11.047421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.048862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.058327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.062400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 293/582] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:11.063307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.077828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.078450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.078738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.079007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] PATH_INFO: `/v3/users/glance` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.087338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.088803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:11.101073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:11.102005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 290/583] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:10 2022] GET /identity/v3/users/glance => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:11.102916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.172546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1f8d68-6b38-45b2-8a8a-b853ab750d4f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.191419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.211458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.212085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.212305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa44bc5-f1bf-4cac-8ac5-393cf9845feb None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.212408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.212620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.213251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:11.213552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.214909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.226984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 294/584] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:11.229061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.248759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.249380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.249761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.250038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.260894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.262536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:11.280025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:11.281242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:11.282317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:11.282317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:11.282317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:11.282317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 291/585] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:11.282317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.319333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.346661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65286dbc-0508-4df4-8b77-f40cdf948557 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.382212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.382874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.383155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.383427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/603c81d781754201a5ac5b7d9b84fab6/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.400546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2d23a7-50ae-4930-b1e2-91ec1cc7f100 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.400767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=603c81d781754201a5ac5b7d9b84fab6, role_id=cb16002d0a2b4241b1d9c4c8ecd9ba42)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.401995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.416237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 295/586] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:11 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/603c81d781754201a5ac5b7d9b84fab6/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:11.417176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.448166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.450193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.451048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.452228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.452794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.454592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:11.455135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.456992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.479075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 292/587] 10.222.0.42 () {56 vars in 1137 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:11.480465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.533233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a79c61-1636-4272-a547-49a275f425af None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.571060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.571808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.572199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.572541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.580696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.582256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:11.596484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:11.597523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:11.598444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 296/588] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/users/cinder => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:11.599022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.604801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.616402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf35b99-f2f6-4bb0-ab65-0f31241a3764 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.658363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.658976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.659265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.659604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.661134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:11.661134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.661959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.675340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 293/589] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:11.676449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.705251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.718580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49d24d1-6f94-490c-ad8c-b8c55128d493 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.753315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.753943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.754221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.754494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.768198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.769659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:11.786504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:11.787705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:11.788818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:11.788818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:11.788818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:11.788818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 297/590] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:11.788818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.812043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e85646f-fda3-4fde-989c-074a4ef76a2c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.854662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.855270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.855600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.855878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.856648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:11.856952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.858699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.879139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 294/591] 10.222.0.42 () {56 vars in 1137 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:11.880050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:11.894183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.935244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3293f6b2-8164-438d-b917-306b23a84f28 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:11.971180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:11.971812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:11.972143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:11.972413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:11.980168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:11.981596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:11.984679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:11.993999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:11.995197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:11.996190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 298/592] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/users/neutron => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:11.996814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.003801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3f5812-006f-4baa-a6ae-4ddc163a36b0 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.042924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02654034-34a0-490c-96a5-42a6cfe3bf0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.043548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02654034-34a0-490c-96a5-42a6cfe3bf0a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.043714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02654034-34a0-490c-96a5-42a6cfe3bf0a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.044025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02654034-34a0-490c-96a5-42a6cfe3bf0a None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.044671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02654034-34a0-490c-96a5-42a6cfe3bf0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:12.044994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02654034-34a0-490c-96a5-42a6cfe3bf0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.046352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02654034-34a0-490c-96a5-42a6cfe3bf0a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.062412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 295/593] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:12.063823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f4f403-a33f-4847-a19c-191ff04eed4f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.064238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f4f403-a33f-4847-a19c-191ff04eed4f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.064607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f4f403-a33f-4847-a19c-191ff04eed4f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.066175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 296/594] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:12.066964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f4f403-a33f-4847-a19c-191ff04eed4f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.110018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757495d4-fc72-482c-922c-4b64510969e0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.145356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:12.145563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.146163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.146433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.146759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.157568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.159002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:12.174458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:12.175438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:12.176707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:12.176707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:12.176707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:12.176707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 299/595] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:12.176707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.184859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f4f403-a33f-4847-a19c-191ff04eed4f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.220927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33260be0-a5e8-4649-9afb-a73c02a96c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.221531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33260be0-a5e8-4649-9afb-a73c02a96c48 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.221802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33260be0-a5e8-4649-9afb-a73c02a96c48 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.222069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33260be0-a5e8-4649-9afb-a73c02a96c48 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.222786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33260be0-a5e8-4649-9afb-a73c02a96c48 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:12.223085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33260be0-a5e8-4649-9afb-a73c02a96c48 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.224549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33260be0-a5e8-4649-9afb-a73c02a96c48 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.232393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:12.244251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 297/596] 10.222.0.42 () {56 vars in 1139 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:12.245814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63510e75-1169-482b-a8e2-3b1e55d025a2 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.246303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63510e75-1169-482b-a8e2-3b1e55d025a2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.246745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63510e75-1169-482b-a8e2-3b1e55d025a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.265698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-63510e75-1169-482b-a8e2-3b1e55d025a2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:12.267167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-63510e75-1169-482b-a8e2-3b1e55d025a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:12.288578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44a238b-b278-4ff8-88ba-039f3c5dd951 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.324699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.325259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.325512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.325876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.332902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.334491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.343996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 298/597] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:12.343996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63510e75-1169-482b-a8e2-3b1e55d025a2 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:12.347208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:12.348064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:12.348965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 300/598] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:12.354365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.417190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:12.469225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63510e75-1169-482b-a8e2-3b1e55d025a2 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.481861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71787e57-311b-4b46-ade4-81612e76068b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.499635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:12.507505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.508165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.508450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.508725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.509829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:12.510148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.512129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.525794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 299/599] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:12.526958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.527722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.528561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.529010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.529769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.544583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.548499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:12.568027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:12.569180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:12.570252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:12.570252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:12.570252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:12.576712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 301/600] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:12.577854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.643709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0daa64-a53a-4b6a-a0f4-52e1d13b3367 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.661107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:12.681953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.682561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.682839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.683110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.683793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:12.684123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.685562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.694242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240757c9-2820-434e-b7b8-e575b385b36d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.702992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 300/601] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:12.702992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.733371 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.734000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.734278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.734550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.741452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.742927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:12.755391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:12.756317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:12.757187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 302/602] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:12.757661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.786011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:12.815688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20c4931-b1fa-4dab-8259-87daa84225b4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.852909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.853762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.854186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.854607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.855455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:12.855980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:12.858074 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:12.870413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273fb291-683a-4d3c-95cf-f564792a5d9e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:12.870889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 301/603] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:12.871925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.910720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:12.911336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:12.911612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:12.911879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:12.918044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:12.923735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:12.929068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:12.930183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:12.931162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 303/604] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:12.935787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:12.987763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074401a7-517d-425d-b296-9b8f1a0787da None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.022306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:13.024456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d1372-1183-4596-bb27-9888747a662b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.025078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d1372-1183-4596-bb27-9888747a662b None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.025352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d1372-1183-4596-bb27-9888747a662b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.025620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d1372-1183-4596-bb27-9888747a662b None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/30b10ebdc87a4747bef7fc60b3cec9c8/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.040044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2705d7f8-098b-42e6-bd0d-90d089b49a87 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.041983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552d1372-1183-4596-bb27-9888747a662b None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=30b10ebdc87a4747bef7fc60b3cec9c8, role_id=cb16002d0a2b4241b1d9c4c8ecd9ba42)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.043407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552d1372-1183-4596-bb27-9888747a662b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.057905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 302/605] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:12 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/30b10ebdc87a4747bef7fc60b3cec9c8/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:13.058966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d1372-1183-4596-bb27-9888747a662b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.095278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.096104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.096843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.097112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.098156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:13.098450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.099925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.119555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 304/606] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:13.120812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.167460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:13.184659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d1372-1183-4596-bb27-9888747a662b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.224445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.225187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.225541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.225968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.234829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.236471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:13.249251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:13.250198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:13.251065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:13.255675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 303/607] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:13.256704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.266479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b846a1b9-6b55-4a4b-a8b6-8dae30ba4ce7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.311695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.312740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.313018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.313282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.314260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:13.314720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.316130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.330603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 305/608] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:13.331521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.345982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:13.385518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98789d4-1710-43a4-a81f-a1913b0de98b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.424021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.424962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.425381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.425801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/d7b63222dc2846f5a42208877b0de123/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.442336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=d7b63222dc2846f5a42208877b0de123, role_id=cb16002d0a2b4241b1d9c4c8ecd9ba42)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.444147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.455430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:13.457795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229b71-9ea9-40ce-a9bc-98e82ccbd40e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.463396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 304/609] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:13 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/d7b63222dc2846f5a42208877b0de123/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:13.464624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.497929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.499116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.499577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.500051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.500922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:13.501403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.503015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.517623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 306/610] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:13.523598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.588951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80386295-9244-4fec-b3b5-07de5b208b2e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.604947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:13.631969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.632800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.633153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.633668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.647718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.649421 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:13.667470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:13.668507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:13.669396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:13.669396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:13.669396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:13.669396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 305/611] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:13.677821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2383f83d-2b3d-4ed9-9cec-9d3baff788cc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.681376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.725366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3e2c47-1cd2-4fc2-844b-119cf6569f0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.728280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3e2c47-1cd2-4fc2-844b-119cf6569f0f None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.728879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3e2c47-1cd2-4fc2-844b-119cf6569f0f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.729391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3e2c47-1cd2-4fc2-844b-119cf6569f0f None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/6a343a260fa2415e84caf959933a6c13/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.752936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3e2c47-1cd2-4fc2-844b-119cf6569f0f None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=6a343a260fa2415e84caf959933a6c13, role_id=cb16002d0a2b4241b1d9c4c8ecd9ba42)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.754763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3e2c47-1cd2-4fc2-844b-119cf6569f0f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.771442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:13.772167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 307/612] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:13 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/6a343a260fa2415e84caf959933a6c13/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:41:13.773722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562b61d-6aff-4ce0-961f-4e0110d08640 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.774596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562b61d-6aff-4ce0-961f-4e0110d08640 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.775072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562b61d-6aff-4ce0-961f-4e0110d08640 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.781666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 308/613] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:13 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:13.797237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24163e1f-de75-4b46-8299-bf947e1a0bcb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.797559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24163e1f-de75-4b46-8299-bf947e1a0bcb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.797778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24163e1f-de75-4b46-8299-bf947e1a0bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.820404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-24163e1f-de75-4b46-8299-bf947e1a0bcb None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:13.821644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-24163e1f-de75-4b46-8299-bf947e1a0bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:13.850798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af8b37c0-2ec4-43f1-a35e-50f142590459 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:13.890079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:13.891100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:13.891573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:13.892117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:13.892976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:13.893468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:13.895099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:13.900862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 309/614] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:41:13.918839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 306/615] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:13.925231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24163e1f-de75-4b46-8299-bf947e1a0bcb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:13.980326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:14.038221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24163e1f-de75-4b46-8299-bf947e1a0bcb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:14.078017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:14.078797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:14.079174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:14.079581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:14.087573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:14.089166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:14.102797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:14.103828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:14.104799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 310/616] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:13 2022] GET /identity/v3/users/swift => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:14.110128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:14.257214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:14.261005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c76383c-874d-4878-ad5b-b7f07bc6d4c7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:14.276311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:14.302001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:14.303006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:14.303491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:14.303984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:14.305314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:14.305861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:14.307696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:14.346689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 307/617] 10.222.0.42 () {56 vars in 1135 bytes} [Fri Sep 16 01:41:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:14.348238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:14.470980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d9ddaf-cec1-4c44-b83d-aee8bd6a8a72 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:14.490122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb5e716-2d92-4fb9-9f0f-0251e5225c38 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:14.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:14.507401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:14.507624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:14.507890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:14.515601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:14.519375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:14.520857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:14.537481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:14.538476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:14.539331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:14.539331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:14.539331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:14.539331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 308/618] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:14.552997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:14.554561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:14.555198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:14.555472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:14.555755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:14.569282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:14.574644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:14.575709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:14.576645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 311/619] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:14.582959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:14.600512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:14.763447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f1f123-344f-40ca-b761-79b650b4229d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:14.797885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0b8e6c-53e3-4637-947e-08b9e5e00a96 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:14.800956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:14.801567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:14.801843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:14.802111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:14.802757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:14.803058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:14.804962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:14.833462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 309/620] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:14.837222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:14.856623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:14.857225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:14.858060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:14.858335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:14.859735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:14.860066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:14.862805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:14.878416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 312/621] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:14.884128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:14.933317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:14.993986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:15.031208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642398c3-1d1e-4cef-911d-4802a8b35ca4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:15.074601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3b0208-4146-40f3-9db9-b27585bb3754 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:15.103231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:15.103912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.104215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.104482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.116764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:15.118205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:15.140800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:15.142644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:15.145165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 310/622] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:15.146304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:15.146304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.146304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.146304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.150008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:15.166357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:15.168098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:15.190669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:15.191647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:15.192561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:15.192561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:15.192561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:15.194626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 313/623] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:15.200884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:15.337514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:15.340045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed12902b-ec6b-4c0f-a8b7-0755abd170cc None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:15.371345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da43ecd-77b3-462c-84c3-ce77b0f46aa5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:15.408549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:15.409163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.409446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.409709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.410403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:15.410702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:15.413232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:15.428305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:15.435094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 311/624] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:15.438722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:15.439345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.439626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.439902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.440592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:15.440894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:15.442082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:15.443669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:15.473226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 314/625] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:15.480273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:15.616906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fa954-cfef-475c-958c-c86aa434d210 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:15.629632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589fc50f-359c-4528-aa6f-b45ed842c38f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:15.662838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975ed3ef-0654-46b7-ba17-a240b1e7f862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:15.663491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ed3ef-0654-46b7-ba17-a240b1e7f862 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.663768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ed3ef-0654-46b7-ba17-a240b1e7f862 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.664071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ed3ef-0654-46b7-ba17-a240b1e7f862 None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/af7a48729497436784b988c6291cfa5d/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.671684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554308bf-2348-4ec2-93b0-40d1416dfd21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:15.672629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554308bf-2348-4ec2-93b0-40d1416dfd21 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.673140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554308bf-2348-4ec2-93b0-40d1416dfd21 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.673605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554308bf-2348-4ec2-93b0-40d1416dfd21 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.681713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975ed3ef-0654-46b7-ba17-a240b1e7f862 None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=af7a48729497436784b988c6291cfa5d, role_id=cb16002d0a2b4241b1d9c4c8ecd9ba42)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:15.683104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554308bf-2348-4ec2-93b0-40d1416dfd21 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:15.683429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975ed3ef-0654-46b7-ba17-a240b1e7f862 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:15.685018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554308bf-2348-4ec2-93b0-40d1416dfd21 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:15.689489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:15.703974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 312/626] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:15 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/af7a48729497436784b988c6291cfa5d/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:15.703974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634ef5b-efc3-4033-97fc-538b90ad46ab None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.703974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634ef5b-efc3-4033-97fc-538b90ad46ab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.703974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c634ef5b-efc3-4033-97fc-538b90ad46ab None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.703974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 313/627] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-554308bf-2348-4ec2-93b0-40d1416dfd21 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:15.707001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:15.707965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:15.709024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 315/628] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:41:15 2022] GET /identity/v3/users/nova => generated 82 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:15.712868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c634ef5b-efc3-4033-97fc-538b90ad46ab None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:15.728742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e98e1c-b41b-4b8a-9717-c4135821fcb7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.729079 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e98e1c-b41b-4b8a-9717-c4135821fcb7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.729339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e98e1c-b41b-4b8a-9717-c4135821fcb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.769539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c3e98e1c-b41b-4b8a-9717-c4135821fcb7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:15.770810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c3e98e1c-b41b-4b8a-9717-c4135821fcb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:15.843729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:15.924793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 316/629] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:15.939338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c634ef5b-efc3-4033-97fc-538b90ad46ab None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:15.975808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af9b332-483f-4b4b-a96b-8ba86ecf497d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:15.976496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af9b332-483f-4b4b-a96b-8ba86ecf497d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:15.976771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af9b332-483f-4b4b-a96b-8ba86ecf497d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:15.977084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af9b332-483f-4b4b-a96b-8ba86ecf497d None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:15.977820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af9b332-483f-4b4b-a96b-8ba86ecf497d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:15.978119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af9b332-483f-4b4b-a96b-8ba86ecf497d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:15.979532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af9b332-483f-4b4b-a96b-8ba86ecf497d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:15.999749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 314/630] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:41:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:16.004967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e98e1c-b41b-4b8a-9717-c4135821fcb7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:16.117388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e98e1c-b41b-4b8a-9717-c4135821fcb7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:16.153709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:16.164287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:16.164994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.165271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.165578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.213920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5c0a9-6d4f-45d0-adae-3a9f1c0243a8 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.214407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5c0a9-6d4f-45d0-adae-3a9f1c0243a8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.214634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c5c0a9-6d4f-45d0-adae-3a9f1c0243a8 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.217380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 315/631] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:16.225110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e977e3-ae9c-4a44-8ad1-7d9770686fc5 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.225402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e977e3-ae9c-4a44-8ad1-7d9770686fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.225679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e977e3-ae9c-4a44-8ad1-7d9770686fc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.254634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d2e977e3-ae9c-4a44-8ad1-7d9770686fc5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:16.256315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d2e977e3-ae9c-4a44-8ad1-7d9770686fc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:16.272833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:16.274812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:16.332227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 316/632] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:16.333501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e977e3-ae9c-4a44-8ad1-7d9770686fc5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:16.377997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:16.414730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 317/633] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:41:16.420485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:16.445310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e977e3-ae9c-4a44-8ad1-7d9770686fc5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:16.487279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:16.488319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.488820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.489309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.500543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:16.505787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:16.506721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:16.507558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:16.510191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 317/634] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:16.515706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:16.518180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:16.625267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc1e4d8-8a79-41b5-97d1-2304a5b70bc1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:16.628792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:16.632977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9092753f-0c20-43d8-9ba8-ba9d70b534c3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:16.666735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:16.667363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.667648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.667914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.669316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cfd3a2-3259-4618-9314-60c333032db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:16.670447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cfd3a2-3259-4618-9314-60c333032db1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.670744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cfd3a2-3259-4618-9314-60c333032db1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.671031 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cfd3a2-3259-4618-9314-60c333032db1 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.671660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cfd3a2-3259-4618-9314-60c333032db1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:16.671985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cfd3a2-3259-4618-9314-60c333032db1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:16.673349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cfd3a2-3259-4618-9314-60c333032db1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:16.679222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:16.680713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:16.685186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 318/635] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:16.685186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fc1112-d67f-468e-946c-21040f473028 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.685186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fc1112-d67f-468e-946c-21040f473028 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.685186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fc1112-d67f-468e-946c-21040f473028 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.685186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 319/636] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:16.686664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fc1112-d67f-468e-946c-21040f473028 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:16.700864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:16.702061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:16.702947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:16.702947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:16.702947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:16.706836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 318/637] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:16.707525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:16.799086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fc1112-d67f-468e-946c-21040f473028 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:16.811959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:16.834443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f76286-d79d-4a53-831f-c0556c933d50 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:16.834834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3c9b22-0819-4c22-b567-bb5c3809ce93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:16.835469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c9b22-0819-4c22-b567-bb5c3809ce93 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.835749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c9b22-0819-4c22-b567-bb5c3809ce93 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.836121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3c9b22-0819-4c22-b567-bb5c3809ce93 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.848899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3c9b22-0819-4c22-b567-bb5c3809ce93 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-7e3c9b22-0819-4c22-b567-bb5c3809ce93 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:16.854413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:16.855623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:16.856500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 320/638] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:16.857181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794f0ef0-fd36-4d71-986b-554721826955 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.857484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794f0ef0-fd36-4d71-986b-554721826955 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.857817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794f0ef0-fd36-4d71-986b-554721826955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.876670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-794f0ef0-fd36-4d71-986b-554721826955 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:16.876962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:16.877715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:16.877961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-794f0ef0-fd36-4d71-986b-554721826955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:16.878153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:16.878308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:16.890491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:16.891955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:16.894106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:16.907320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:16.908277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:16.909161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:16.909161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:16.909161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:16.909161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 319/639] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:16.909985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:16.957508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 321/640] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:16.958352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794f0ef0-fd36-4d71-986b-554721826955 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.055476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e755333a-761f-4db2-a693-c67c7cc68634 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.072168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794f0ef0-fd36-4d71-986b-554721826955 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.103149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.105000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.105277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.105541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.106181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:17.106481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.107863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.109538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.118370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.120462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.121012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.121440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.122271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:17.122729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.124895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.127920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 320/641] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:17.128920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.141535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 322/642] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:17.141535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.162957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.241239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb13c46-3578-4a6a-b588-76d92b4d0948 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.273103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a05da58-958c-41c7-a8e4-092d9c4821eb None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.278987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.281114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.281542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.281952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.283096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:17.283554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.285673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.303422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 321/643] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:17.304704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.310202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.310806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.311082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.311351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.318304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.319745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:17.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:17.333253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 323/644] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:17.334848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.374815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.414843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.426881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba7aafa-42b7-4f8c-b868-f93a7510018a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.447490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0595317-3471-4d6a-b0e1-018229d71c8b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.460244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.460862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.461128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.461401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.472454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.473923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.488307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:17.488561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:17.489882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:17.491166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:17.491166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:17.491166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:17.491166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 322/645] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:17.491166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.492259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.492686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.493079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.505894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:17.512102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:17.513199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:17.514224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:17.516300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 324/646] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:17.517227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.598387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add3b70-fd0b-45a2-9194-b0d93fee0692 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.605921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.623878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79fe3ff-ff61-4490-98a9-029f36fa135d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.634856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.635485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.635761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.636053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] PATH_INFO: `/v3/users/glance` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.644038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.645474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.655432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:17.657269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:17.658623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 323/647] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/users/glance => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:17.659562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.666424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.667041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.667317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.667605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.668295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:17.668594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.670075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.688135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 325/648] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:17.688135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.765458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdaaff4-47d1-4d03-bb4a-9e154971c28b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.792116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad770432-722f-4fce-aace-c4f0261d3a34 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.804987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.806206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.806688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.807338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.808600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:17.809868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.812151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.827048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 324/649] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:17.827869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.834566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.835182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.835462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.835731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.836566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:17.836922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.838434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:17.852747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 326/650] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:17.852747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:17.859090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.916774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:17.936425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ba40f9-6c3b-4227-b3f6-69174e8ce7fb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.965009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6cd24c-72bf-4d15-aca0-43ce1d0e9399 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:17.977409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:17.978004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:17.978403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:17.978615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:17.979330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:17.979869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:17.981135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.005857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 325/651] 10.222.0.42 () {56 vars in 1137 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:18.005857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.006953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.007555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.007828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.008117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.009047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=603c81d781754201a5ac5b7d9b84fab6) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:18.010186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.011871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.027963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 327/652] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:17 2022] GET /identity/v3/role_assignments?user.id=603c81d781754201a5ac5b7d9b84fab6&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:18.027963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.129292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.153893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3f1293-d470-467d-9ed4-f7425cef1964 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:18.166200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ca618-27f7-429f-98d3-3bfca3162ad9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:18.185564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.205334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2f2a5c-2e5e-4e92-80a8-64aa37cdc63d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.205788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.205976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f2a5c-2e5e-4e92-80a8-64aa37cdc63d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.206243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f2a5c-2e5e-4e92-80a8-64aa37cdc63d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.206426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.206519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f2a5c-2e5e-4e92-80a8-64aa37cdc63d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.206670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.207012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.215460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.216829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.218688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2f2a5c-2e5e-4e92-80a8-64aa37cdc63d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.220678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2f2a5c-2e5e-4e92-80a8-64aa37cdc63d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:18.230187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:18.231147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 326/653] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/users/cinder => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:18.232163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-9f2f2a5c-2e5e-4e92-80a8-64aa37cdc63d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:18.243727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:18.244969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:18.245858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:18.245858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:18.245858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:18.245858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 328/654] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:18.246528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981fc1db-800b-46f4-8551-59d5ca118c70 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.246626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981fc1db-800b-46f4-8551-59d5ca118c70 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.247105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981fc1db-800b-46f4-8551-59d5ca118c70 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.248841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 329/655] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:18.251662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981fc1db-800b-46f4-8551-59d5ca118c70 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.346197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abc8516-694d-4236-a382-e1953e6f9595 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:18.368250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981fc1db-800b-46f4-8551-59d5ca118c70 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:18.382026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb24092-bc06-4679-8330-51b682a3cf69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.382871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb24092-bc06-4679-8330-51b682a3cf69 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.382871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb24092-bc06-4679-8330-51b682a3cf69 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.383099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb24092-bc06-4679-8330-51b682a3cf69 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.399140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.407422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.408055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.408641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.408641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.409312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:18.409608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.411014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.429125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 330/656] 10.222.0.42 () {56 vars in 1137 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:18.429961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.459846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.462613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb24092-bc06-4679-8330-51b682a3cf69 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.464229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb24092-bc06-4679-8330-51b682a3cf69 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.539357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc4a40-85a8-44c8-a656-d08854ea5c75 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:18.573203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 327/657] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:18.573203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744670c5-faed-464e-bb04-3e0a545e1ee3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.573203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744670c5-faed-464e-bb04-3e0a545e1ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.573203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744670c5-faed-464e-bb04-3e0a545e1ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.583355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.583993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.584274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.584541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.585268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:18.585567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.586965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.589387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-744670c5-faed-464e-bb04-3e0a545e1ee3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:18.590586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-744670c5-faed-464e-bb04-3e0a545e1ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:18.603263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 331/658] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:18.603263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.638261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.672111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 328/659] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:18.672111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744670c5-faed-464e-bb04-3e0a545e1ee3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.705341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.719480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb18ffc2-8df2-464c-8e9e-bda9eccf0d96 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:18.759968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.760570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.760850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.761226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.783384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744670c5-faed-464e-bb04-3e0a545e1ee3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:18.824888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:18.826619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:18.827048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:18.827468 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:18.840538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.842906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.855231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:18.857120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:18.862774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:18.863746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:18.864662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:18.864662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:18.864662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:18.867121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 329/660] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:18.868174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.906585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.962784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:18.984987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 332/661] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:18.984987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:18.991486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29046cd3-48ab-438c-827c-e8dbcd311edf None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.027144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.027768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.028072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.028337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.036503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.037956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:19.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:19.049932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 330/662] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/users/neutron => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:19.050877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.095766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2512af83-a0d3-49c9-ae37-78e7295b8898 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.136870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.137773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.138211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.139046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.139689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:19.140005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.141831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.142546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.156113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 333/663] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:19.156989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.160874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07fa1f2-921b-4384-8d9b-1d40685c2c2e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.195190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.196087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.197131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.197416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.209560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:19.211633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:19.215496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:19.220540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 331/664] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:19.221358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.274121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b3f87d-10dc-4ae2-992c-efc2447c17ea None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.310002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.310677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.310957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.311220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.324520 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.325591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac478fd-1c99-4901-843a-7a5cb0042ca1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.325956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:19.343265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:19.344262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:19.345122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:19.345122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:19.345122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:19.349029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 334/665] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:19.349624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.364965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.365717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.365992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.366336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.367123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:19.367424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.368882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.390890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 332/666] 10.222.0.42 () {56 vars in 1139 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:19.391720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.399365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.456741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.460257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3a3571-0346-4412-869d-2d3fefb64953 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.499827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.500486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.500748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.501013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.502507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4e8dff-f902-4e1f-a5bf-d33f564095e5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.508361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.509890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:19.522658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:19.523532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 335/667] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:19.524774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.543748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.544579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.545087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.545558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.546217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:19.546515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.547996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.560399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 333/668] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:19.560399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.624210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.642204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715feeb8-ee77-46e9-a325-4beb16cc8003 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.662975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6565c1ca-dadc-4dc6-ac2e-3d2322a21a4b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.676405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.677005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.677497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.677651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.678274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:19.678578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.680019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.691928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 336/669] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:19.696570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.704664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.705792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.706168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.706566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.719055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.776133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.777581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.808294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75397ddd-f4be-4681-8e3a-d15777732220 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.848053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:19.848804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:19.849183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:19.849578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:19.849933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.850446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:19.850853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:19.852407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:19.864867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 337/670] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:19.870353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.894487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 334/671] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:19.894487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:19.964187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:19.986667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f463d4-ecd6-4fb8-92e1-09a896d92718 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:19.990409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e00982-e56c-4a94-be4b-9873080189c9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.025339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.026053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.026398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.026725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.026984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.027670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.028039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.028377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.034808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.036365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.041240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.042717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:20.048304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:20.049350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:20.050524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:20.051215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 335/672] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:20.052120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:20.065089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:20.066075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:20.066962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:20.066962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:20.066962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:20.066962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 338/673] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:20.066962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.085668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:20.169532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b117e03b-f850-4579-bcdf-4e99b512ead3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.174127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e075ca73-4dbc-47a4-bebe-16e6e363d3e6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.208203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:20.208914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.210100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.210525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.210866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.211661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=30b10ebdc87a4747bef7fc60b3cec9c8) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:20.212082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.213516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.213853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.214846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.215615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.215955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.227977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 336/674] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/role_assignments?user.id=30b10ebdc87a4747bef7fc60b3cec9c8&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:20.227977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.235426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.237232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:20.257784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:20.258980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:20.260014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:20.260014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:20.260014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:20.260014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 339/675] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:20.260900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.348333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:20.359805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df01a73c-2c6f-42b0-8820-8252fc5fe27a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.380943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3ce0f-b3f2-4bfe-88b1-7f8c6bb24194 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.401728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.402331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.402609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.402883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.403571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:20.403873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.405421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.420838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 337/676] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:20.421055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.429287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08643357-99cd-4ae5-b895-2dd23e68e546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.430171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08643357-99cd-4ae5-b895-2dd23e68e546 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.430622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08643357-99cd-4ae5-b895-2dd23e68e546 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.431073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08643357-99cd-4ae5-b895-2dd23e68e546 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.431814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08643357-99cd-4ae5-b895-2dd23e68e546 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:20.432162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08643357-99cd-4ae5-b895-2dd23e68e546 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.433516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08643357-99cd-4ae5-b895-2dd23e68e546 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.446244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 340/677] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:20.447699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec924f1-48ba-4ab7-b710-33e909ca82c2 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.448362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec924f1-48ba-4ab7-b710-33e909ca82c2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.448776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec924f1-48ba-4ab7-b710-33e909ca82c2 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.450517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 341/678] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:20.452126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec924f1-48ba-4ab7-b710-33e909ca82c2 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.509358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:20.535814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31016e4-d8ea-4d18-91f7-dcad111af10b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.572681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.573281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.573593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.573824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.574189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec924f1-48ba-4ab7-b710-33e909ca82c2 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.574463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:20.574761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.576411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.590561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 338/679] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:20.591388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.607080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:20.614623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b616fb8-bafd-4532-8158-f197e968a936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.615234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b616fb8-bafd-4532-8158-f197e968a936 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.615514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b616fb8-bafd-4532-8158-f197e968a936 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.615785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b616fb8-bafd-4532-8158-f197e968a936 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.616551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b616fb8-bafd-4532-8158-f197e968a936 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=d7b63222dc2846f5a42208877b0de123) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:20.616857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b616fb8-bafd-4532-8158-f197e968a936 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.618280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b616fb8-bafd-4532-8158-f197e968a936 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.631971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 342/680] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/role_assignments?user.id=d7b63222dc2846f5a42208877b0de123&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:20.631971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701358d1-98ea-497e-851c-6ee2274e6058 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.631971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701358d1-98ea-497e-851c-6ee2274e6058 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.631971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701358d1-98ea-497e-851c-6ee2274e6058 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.649961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-701358d1-98ea-497e-851c-6ee2274e6058 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:20.651201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-701358d1-98ea-497e-851c-6ee2274e6058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:20.743552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a90956-fee8-4c47-8de5-7e5c425f3bbb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.744136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 343/681] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:20.745594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701358d1-98ea-497e-851c-6ee2274e6058 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.793372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.793972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.794255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.794532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.804127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:20.814404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.817580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:20.832446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:20.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:20.834335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 339/682] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:20.839699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.896618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701358d1-98ea-497e-851c-6ee2274e6058 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:20.937354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:20.937868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:20.938573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:20.939098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:20.939429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:20.947112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:20.948595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:20.961586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:20.962492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:20.963348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 344/683] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:20.970207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:20.975990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5ff66a-61d4-4c30-9fbe-44c52ad4e8b6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.021248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.022290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.022571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.023122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.023863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:21.024200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:21.026358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:21.050922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 340/684] 10.222.0.42 () {56 vars in 1135 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:21.056572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.084840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:21.089469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568ed3e2-5886-472b-b041-e0c9303bffa1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.127222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.127822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.128127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.128395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.129078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:21.129378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:21.130788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:21.144033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 345/685] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:21.144929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.178550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670084dc-7c2f-47c9-9609-7c230903d816 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.215190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:21.228420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.229075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.229385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.230088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.254410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c4060d1-d3b4-47d5-8ea2-5525ca38937a None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.303901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.305105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.305554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.306412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.315899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:21.317607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:21.321757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:21.328245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:21.329347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:21.330364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:21.333501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 346/686] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:21.334799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.362275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:21.437059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 341/687] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 16 01:41:21.438132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.474495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbab4cd-d6d1-4a58-8582-5e3eb7c1ab9f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.480394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:21.522792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.524169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.524599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.525016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.526157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=6a343a260fa2415e84caf959933a6c13) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:21.526610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:21.528233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:21.537414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 347/688] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/role_assignments?user.id=6a343a260fa2415e84caf959933a6c13&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 608 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:21.538974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.546651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb28ce4-5b0d-4bcc-af24-4f602cf5c299 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.629585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.630256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.630541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.630812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.631493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:21.631866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:21.633304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:21.655069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 342/689] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:21.658540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.698136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:21.736864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ff57d-8cf8-47b0-a325-da0329629613 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.783405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:21.795361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36963d9e-3a06-406f-b97f-79d723c766eb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:21.798651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.798651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.798651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.798651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.814277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:21.816145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:21.834932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:21.835966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:21.836871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:21.836871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:21.836871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:21.840014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 348/690] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:21.842177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.852998 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:21.853942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:21.854434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:21.854912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:21.868468 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:21.869927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:21.890523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:21.891561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:21.892556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:21.892556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:21.892556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:21.892983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 343/691] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:21.901012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:21.981557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a1abd8-a97d-4239-8448-88c8623842e3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.042045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:22.042907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.043336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.044359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.045232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:22.045715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:22.047219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:22.049145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:22.062804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 349/692] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:22.068095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:22.077924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e25436-234d-4628-b287-89a9a105e8ce None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.149631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:22.150157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.150454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.150747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.151342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:22.151639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:22.153036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:22.182377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 344/693] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:22.188599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:22.191488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462470c9-a4ce-4989-8b70-aa97014d08af None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.195358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:22.231519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:22.232738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.233481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.233958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.241784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:22.243591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:22.256412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:22.257323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:22.258186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:22.260777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 350/694] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:22.269195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:22.366296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:22.405363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1412b2-995d-491b-bc23-7ad25b0d270e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.421714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5156b2f-6f80-4ae7-8056-a414fbb3579d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.463372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:22.464014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.465057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.465984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.466911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:22.467336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:22.472642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:22.478642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:22.479295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.479571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.479911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.496462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:22.498028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:22.504824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 351/695] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:22.520539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:22.521569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:22.523234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 345/696] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:41:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:22.523841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:22.527761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:22.612293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:22.680244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74b89e5-412c-4737-9d9c-788d91d15bea None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.693027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82204c1-3869-4cd5-8755-64cc446b1bc4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.712185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:22.719050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10743ba-0fcf-436f-9f94-62a4878819aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:22.719789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10743ba-0fcf-436f-9f94-62a4878819aa None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.720189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10743ba-0fcf-436f-9f94-62a4878819aa None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.720552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10743ba-0fcf-436f-9f94-62a4878819aa None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.721481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10743ba-0fcf-436f-9f94-62a4878819aa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:22.721881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10743ba-0fcf-436f-9f94-62a4878819aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:22.723414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10743ba-0fcf-436f-9f94-62a4878819aa None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:22.736799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66953df3-8c78-409f-8eac-0f7ac5510438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:22.737412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66953df3-8c78-409f-8eac-0f7ac5510438 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.737690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66953df3-8c78-409f-8eac-0f7ac5510438 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.737957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66953df3-8c78-409f-8eac-0f7ac5510438 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.738688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66953df3-8c78-409f-8eac-0f7ac5510438 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=af7a48729497436784b988c6291cfa5d) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:22.738996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66953df3-8c78-409f-8eac-0f7ac5510438 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:22.741371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66953df3-8c78-409f-8eac-0f7ac5510438 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:22.743409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 346/697] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:41:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:22.744919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13517483-44fa-47d3-b533-c9af78e1179d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.745304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13517483-44fa-47d3-b533-c9af78e1179d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.745670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13517483-44fa-47d3-b533-c9af78e1179d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.751752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 347/699] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:22 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:22.752772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13517483-44fa-47d3-b533-c9af78e1179d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:22.755786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 352/699] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:22 2022] GET /identity/v3/role_assignments?user.id=af7a48729497436784b988c6291cfa5d&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 608 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:22.757303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a37052-383b-4cd7-a3e0-9f073995aa67 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:22.757595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a37052-383b-4cd7-a3e0-9f073995aa67 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:22.757878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a37052-383b-4cd7-a3e0-9f073995aa67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:22.795973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-80a37052-383b-4cd7-a3e0-9f073995aa67 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:22.795973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-80a37052-383b-4cd7-a3e0-9f073995aa67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:22.924745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 353/700] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:22.961657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13517483-44fa-47d3-b533-c9af78e1179d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:22.983960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:23.001006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d51477-be7b-457f-9377-963290404aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:23.001669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d51477-be7b-457f-9377-963290404aff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.001947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d51477-be7b-457f-9377-963290404aff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.002268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d51477-be7b-457f-9377-963290404aff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.075077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d51477-be7b-457f-9377-963290404aff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:23.076625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d51477-be7b-457f-9377-963290404aff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:23.181318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 348/701] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:23.186729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a37052-383b-4cd7-a3e0-9f073995aa67 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:23.224681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d51477-be7b-457f-9377-963290404aff None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:23.251644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:23.320055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a37052-383b-4cd7-a3e0-9f073995aa67 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:23.331747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:23.353375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d51477-be7b-457f-9377-963290404aff None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:23.363083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6141946a-6593-4456-9ea3-4a23f43a5358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:23.363726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6141946a-6593-4456-9ea3-4a23f43a5358 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.364043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6141946a-6593-4456-9ea3-4a23f43a5358 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.364313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6141946a-6593-4456-9ea3-4a23f43a5358 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.377263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6141946a-6593-4456-9ea3-4a23f43a5358 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:23.379100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6141946a-6593-4456-9ea3-4a23f43a5358 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-6141946a-6593-4456-9ea3-4a23f43a5358 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:23.397747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:23.398728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:23.399689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:23.399689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:23.399689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:23.401955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 354/702] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:23.403425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bd402c-b1e3-4838-965a-7056b4dd2c54 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.404027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bd402c-b1e3-4838-965a-7056b4dd2c54 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.404476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bd402c-b1e3-4838-965a-7056b4dd2c54 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.405039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b014e872-c32f-4915-aad9-d9baf1c39835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:23.405740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b014e872-c32f-4915-aad9-d9baf1c39835 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.406091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b014e872-c32f-4915-aad9-d9baf1c39835 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.406580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b014e872-c32f-4915-aad9-d9baf1c39835 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.406934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 355/703] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:23.407814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bd402c-b1e3-4838-965a-7056b4dd2c54 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:23.418469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b014e872-c32f-4915-aad9-d9baf1c39835 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:23.421070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b014e872-c32f-4915-aad9-d9baf1c39835 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b014e872-c32f-4915-aad9-d9baf1c39835 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:23.443975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:23.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:23.445880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:23.445880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:23.445880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:23.445880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 349/704] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:23.452048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e54e0b-0308-4f4d-8fcd-2c329e3b5f09 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.452048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e54e0b-0308-4f4d-8fcd-2c329e3b5f09 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.452048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e54e0b-0308-4f4d-8fcd-2c329e3b5f09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.468510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-02e54e0b-0308-4f4d-8fcd-2c329e3b5f09 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:23.470022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-02e54e0b-0308-4f4d-8fcd-2c329e3b5f09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:23.549093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bd402c-b1e3-4838-965a-7056b4dd2c54 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:23.561158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 350/705] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:23.562293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e54e0b-0308-4f4d-8fcd-2c329e3b5f09 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:23.565327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:23.612164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:23.612890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.613291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.613629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.614815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:23.615184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:23.617064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:23.633602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 356/706] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:23.643877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:23.690890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:23.696393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e54e0b-0308-4f4d-8fcd-2c329e3b5f09 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:23.776690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:23.777303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.777580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.777850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.778496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:23.778822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:23.784701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:23.805305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 351/707] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:23.817186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:23.846289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f7a82-2091-4355-a2a6-ba0775739a72 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:23.903006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:23.903646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:23.903987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:23.904266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:23.922475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:23.925373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:23.952569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:23.953489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:23.954398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:23.957444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 357/708] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:23.959126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:23.965469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:24.015916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97dd90a-65ff-428c-bbfa-2c72a2024546 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:24.053350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:24.053961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.054240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.054508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.055150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:24.056703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:24.074873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:24.088184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 352/709] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:41:23 2022] POST /identity/v3/users => generated 254 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:24.089234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:24.100966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d70532c-f544-49f5-b95d-f79ca98467cd None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:24.155986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:24.157100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.157905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.158704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.159817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:24.160438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:24.162158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:24.178188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 358/710] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:24.185144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:24.338924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e1a45c-7c70-48ce-97b2-9679eb35ead4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:24.349465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c81dd9b-385d-4de3-9d3a-5266d029b9e2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:24.376722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:24.393394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:24.395388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.395876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.397254 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.398245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=603c81d781754201a5ac5b7d9b84fab6) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:24.398750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:24.400750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:24.402772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bacc594-d5dc-436a-8189-1b77335a77a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:24.403426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bacc594-d5dc-436a-8189-1b77335a77a9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.404357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bacc594-d5dc-436a-8189-1b77335a77a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.406158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bacc594-d5dc-436a-8189-1b77335a77a9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.406982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bacc594-d5dc-436a-8189-1b77335a77a9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:24.408389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bacc594-d5dc-436a-8189-1b77335a77a9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:24.411582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 359/711] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:24 2022] GET /identity/v3/role_assignments?user.id=603c81d781754201a5ac5b7d9b84fab6&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-7bacc594-d5dc-436a-8189-1b77335a77a9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:24.425243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:24.426338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:24.427169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:24.430738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 353/712] 10.222.0.42 () {56 vars in 1129 bytes} [Fri Sep 16 01:41:24 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:24.432887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:24.438770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:24.533599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410e662-aad7-4a67-97b0-99bbab22c15b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.534057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410e662-aad7-4a67-97b0-99bbab22c15b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.534606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410e662-aad7-4a67-97b0-99bbab22c15b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.537884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 354/713] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:24.549184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3747e91a-d5b6-4806-9f6a-6176026c8c8c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.549825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3747e91a-d5b6-4806-9f6a-6176026c8c8c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.550322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3747e91a-d5b6-4806-9f6a-6176026c8c8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.573463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3747e91a-d5b6-4806-9f6a-6176026c8c8c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:24.575039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3747e91a-d5b6-4806-9f6a-6176026c8c8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:24.593366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6161c1e-907b-4ace-8c76-7dfc5a51dc73 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:24.658720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:24.659724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.660260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.660735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.661892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:24.662438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:24.664928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:24.671335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 355/715] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:41:24.672014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 360/715] 10.222.0.42 () {56 vars in 1124 bytes} [Fri Sep 16 01:41:24 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:24.683925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3747e91a-d5b6-4806-9f6a-6176026c8c8c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:24.862376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3747e91a-d5b6-4806-9f6a-6176026c8c8c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:24.885862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:24.921855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:24.922461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:24.922735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:24.923001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:24.923615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:24.923909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:24.925288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:24.932896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 356/716] 10.222.0.42 () {56 vars in 1124 bytes} [Fri Sep 16 01:41:24 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:25.017583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:25.096236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:25.138788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656393f7-7d69-4766-8ab0-cc355a483f68 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:25.179459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:25.180137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:25.180421 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:25.180717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] PATH_INFO: `/v3/services/network` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:25.181272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:25.182621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:25.196468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:25.197495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:25.198323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 361/717] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:25 2022] GET /identity/v3/services/network => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:25.203723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:25.359871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42169bf6-2da1-42fb-bd3f-359234f6d0b0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:25.409137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abb3185-46c3-4f4c-96a1-389f3df6ff82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:25.410072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abb3185-46c3-4f4c-96a1-389f3df6ff82 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:25.410556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abb3185-46c3-4f4c-96a1-389f3df6ff82 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:25.411028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abb3185-46c3-4f4c-96a1-389f3df6ff82 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:25.411874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abb3185-46c3-4f4c-96a1-389f3df6ff82 None admin] RBAC: Adding query filter params (name=network) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:25.412420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abb3185-46c3-4f4c-96a1-389f3df6ff82 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:25.414057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abb3185-46c3-4f4c-96a1-389f3df6ff82 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:25.424365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 357/718] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:25.429571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:25.572960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7106a6fb-bad8-46ee-880e-9a757b128630 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:25.621284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:25.628342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c890bef0-bef9-4e4d-97d7-211d1e4d98ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:25.628950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c890bef0-bef9-4e4d-97d7-211d1e4d98ac None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:25.629245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c890bef0-bef9-4e4d-97d7-211d1e4d98ac None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:25.629482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c890bef0-bef9-4e4d-97d7-211d1e4d98ac None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:25.630101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c890bef0-bef9-4e4d-97d7-211d1e4d98ac None admin] RBAC: Adding query filter params (type=network) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:25.630396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c890bef0-bef9-4e4d-97d7-211d1e4d98ac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:25.631750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c890bef0-bef9-4e4d-97d7-211d1e4d98ac None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:25.641682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 362/719] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:25 2022] GET /identity/v3/services?type=network => generated 124 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:25.782401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25ccd5-2ccb-4eb1-83da-22e4eeeddf22 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:25.783040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25ccd5-2ccb-4eb1-83da-22e4eeeddf22 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:25.783526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25ccd5-2ccb-4eb1-83da-22e4eeeddf22 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:25.785450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 358/720] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:25.790796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78e7fc-9711-401b-8da6-f544c2dcf886 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:25.791083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78e7fc-9711-401b-8da6-f544c2dcf886 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:25.791357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78e7fc-9711-401b-8da6-f544c2dcf886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:25.821924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8a78e7fc-9711-401b-8da6-f544c2dcf886 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:25.823141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8a78e7fc-9711-401b-8da6-f544c2dcf886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:25.914703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 363/721] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:26.197435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25ccd5-2ccb-4eb1-83da-22e4eeeddf22 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:26.251470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:26.334321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25ccd5-2ccb-4eb1-83da-22e4eeeddf22 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:26.379160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:26.380636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:26.381151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:26.381620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:26.400413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:26.406314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:26.407288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:26.408180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:26.410788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 359/722] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:26.416189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78e7fc-9711-401b-8da6-f544c2dcf886 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:26.527044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78e7fc-9711-401b-8da6-f544c2dcf886 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:26.556170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:26.573516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:26.574477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:26.574996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:26.575540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:26.576444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:26.576949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:26.578564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:26.588329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 364/723] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:26.594431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:26.726206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:26.733086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189b3685-e5bb-47ce-9dae-adccbc7a23b6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:26.781552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:26.782104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:26.782378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:26.782644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:26.801947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:26.803386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:26.823730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:26.824740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:26.825786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:26.825786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:26.825786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:26.825786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 360/724] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:26.830483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:26.948104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cb0a4a-7c09-4537-8c11-4dc2e130736a None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:26.991514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:26.992513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:26.993042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:26.993528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:26.994372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:26.994873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:26.996545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:27.014449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 365/725] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:27.019724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:27.170487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747b30ab-ce5f-44f7-8002-6d5edea8278a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:27.218283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8507513f-bdfc-4c04-800b-44c9d2f8d33b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:27.218912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8507513f-bdfc-4c04-800b-44c9d2f8d33b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.219184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8507513f-bdfc-4c04-800b-44c9d2f8d33b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.219449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8507513f-bdfc-4c04-800b-44c9d2f8d33b None admin] PATH_INFO: `/v3/users/swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.227199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8507513f-bdfc-4c04-800b-44c9d2f8d33b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:27.228652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8507513f-bdfc-4c04-800b-44c9d2f8d33b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8507513f-bdfc-4c04-800b-44c9d2f8d33b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:27.241048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:27.242037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:27.243002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 361/726] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:27.248604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:27.305172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed482ea-114e-477a-bd90-ed22e89897a0 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.305804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed482ea-114e-477a-bd90-ed22e89897a0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.306291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed482ea-114e-477a-bd90-ed22e89897a0 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.308184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 362/727] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:27.314407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b8d60f-b80e-43ec-bc16-8b47c925c02e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.315297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b8d60f-b80e-43ec-bc16-8b47c925c02e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.316092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b8d60f-b80e-43ec-bc16-8b47c925c02e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.323606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:27.351174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-73b8d60f-b80e-43ec-bc16-8b47c925c02e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:27.352712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-73b8d60f-b80e-43ec-bc16-8b47c925c02e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:27.367794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:27.381104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ac5399-b3fe-45f5-8589-214916273cab None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:27.425225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:27.426626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.427102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.427581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.428965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:27.429461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:27.431834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:27.455243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 363/728] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 16 01:41:27.456570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab74b68-5ce8-45ee-b14c-f3b873c78125 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.456854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab74b68-5ce8-45ee-b14c-f3b873c78125 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.457126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab74b68-5ce8-45ee-b14c-f3b873c78125 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.457727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 366/729] 10.222.0.42 () {56 vars in 1135 bytes} [Fri Sep 16 01:41:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:27.458465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 364/730] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:27.469420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988a701b-38da-49c7-9ac2-b49cb3da81dd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.469704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988a701b-38da-49c7-9ac2-b49cb3da81dd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.469980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988a701b-38da-49c7-9ac2-b49cb3da81dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.472516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:27.524952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-988a701b-38da-49c7-9ac2-b49cb3da81dd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:27.526140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-988a701b-38da-49c7-9ac2-b49cb3da81dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:27.590404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877fdc96-e5ff-4916-806d-3558ecaa5805 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:27.601621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:27.634651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea931e7e-ed67-473b-aa1a-9b8412cca435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:27.635608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea931e7e-ed67-473b-aa1a-9b8412cca435 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.636145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea931e7e-ed67-473b-aa1a-9b8412cca435 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.636853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea931e7e-ed67-473b-aa1a-9b8412cca435 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.704134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 365/731] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:27.714985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea931e7e-ed67-473b-aa1a-9b8412cca435 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:27.716988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea931e7e-ed67-473b-aa1a-9b8412cca435 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:27.743262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988a701b-38da-49c7-9ac2-b49cb3da81dd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:27.820388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:27.844942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 367/732] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 16 01:41:27.846314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad42c9-cefc-4b75-9d12-d19aee73cbb3 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.846603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad42c9-cefc-4b75-9d12-d19aee73cbb3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.846876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ad42c9-cefc-4b75-9d12-d19aee73cbb3 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.848334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 368/733] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:27.848864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad42c9-cefc-4b75-9d12-d19aee73cbb3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:27.860106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:27.864034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988a701b-38da-49c7-9ac2-b49cb3da81dd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:27.908120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0147091d-eff8-4fea-9eb9-63f754a95cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:27.908778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0147091d-eff8-4fea-9eb9-63f754a95cfc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.909057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0147091d-eff8-4fea-9eb9-63f754a95cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.909328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0147091d-eff8-4fea-9eb9-63f754a95cfc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.920228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0147091d-eff8-4fea-9eb9-63f754a95cfc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0147091d-eff8-4fea-9eb9-63f754a95cfc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:27.925146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:27.926260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:27.927292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:27.931076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 366/734] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:27.932346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba722e31-f016-4852-9289-b440b099e4c3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:27.932673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba722e31-f016-4852-9289-b440b099e4c3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:27.932901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba722e31-f016-4852-9289-b440b099e4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:27.951803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ba722e31-f016-4852-9289-b440b099e4c3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:27.953317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ba722e31-f016-4852-9289-b440b099e4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:27.995358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ad42c9-cefc-4b75-9d12-d19aee73cbb3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.024199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 367/735] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:28.025049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba722e31-f016-4852-9289-b440b099e4c3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.041486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.042512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.043093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.043559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.057251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.059378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:28.077864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:28.078813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:28.079686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:28.079686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:28.079686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:28.086015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 369/736] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:28.086015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.098092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:28.140289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba722e31-f016-4852-9289-b440b099e4c3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.157032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:28.187223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.188550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.189066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.189544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.190906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:28.191429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.193534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.203299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a06cd6-de86-447a-968a-936e6e142ad1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.207186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 368/737] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:28.208240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.244495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.245132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.245626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.245852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.256882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:28.262756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:28.263710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:28.264553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 370/738] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:28.265242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.344047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04bf0d2-81ea-437d-9e51-d459d812c3a0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.368289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:28.380938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.381549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.381827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.382096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.382740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:28.383039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.384437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.391122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8428b80e-a9af-438d-a8a1-e45aa0b552a4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.401916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 369/739] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:28.401916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.414615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:28.432313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.432943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.433224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.433492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.445439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.446955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:28.460990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:28.461938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:28.462848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:28.462848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:28.462848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:28.466735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 371/740] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:28.467537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.497150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd0092a-187f-4411-adb2-158c594fab6b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.531054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.531645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.531914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.532208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.532853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:28.533175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.534522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.542365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 370/741] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:28.543160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.565357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3ea01d-0270-4282-98e3-5a4f3719cc8d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.588621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:28.598571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c6f538-418c-427a-9875-d7bb758a0329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.599173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c6f538-418c-427a-9875-d7bb758a0329 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.599451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c6f538-418c-427a-9875-d7bb758a0329 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.599720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c6f538-418c-427a-9875-d7bb758a0329 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.600394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c6f538-418c-427a-9875-d7bb758a0329 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.601744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c6f538-418c-427a-9875-d7bb758a0329 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.618109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 372/742] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:41:28 2022] POST /identity/v3/services => generated 251 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:28.619447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c29f1c-f88d-4943-8e7d-f53d2bc27480 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.619730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c29f1c-f88d-4943-8e7d-f53d2bc27480 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.620032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c29f1c-f88d-4943-8e7d-f53d2bc27480 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.621228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 373/743] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:28.621932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c29f1c-f88d-4943-8e7d-f53d2bc27480 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.658887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:28.659322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e9270-b991-4bca-abe1-f527da4e717c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.715277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.715952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.716798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.717078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.724733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.726310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:28.739474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:28.745728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 371/744] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:28.746824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.762966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c29f1c-f88d-4943-8e7d-f53d2bc27480 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.802583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673baa92-b17b-4bcb-be16-d7fddde0e444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.803199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673baa92-b17b-4bcb-be16-d7fddde0e444 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.803476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673baa92-b17b-4bcb-be16-d7fddde0e444 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.803746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673baa92-b17b-4bcb-be16-d7fddde0e444 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.804429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673baa92-b17b-4bcb-be16-d7fddde0e444 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:28.804740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673baa92-b17b-4bcb-be16-d7fddde0e444 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.806099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673baa92-b17b-4bcb-be16-d7fddde0e444 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.821027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 374/745] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:28.822263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9b279-905b-4062-affc-1e3c6ded2902 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.822552 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9b279-905b-4062-affc-1e3c6ded2902 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.822824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f9b279-905b-4062-affc-1e3c6ded2902 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.841001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f9f9b279-905b-4062-affc-1e3c6ded2902 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:28.842408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f9f9b279-905b-4062-affc-1e3c6ded2902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:28.865826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84318c74-7b42-453e-adb9-dfa653737254 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:28.867763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:28.903325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:28.904533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:28.904822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:28.905061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:28.916878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 375/746] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:28 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 16 01:41:28.917382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:28.918445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f9b279-905b-4062-affc-1e3c6ded2902 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.918554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:28.933427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:28.934476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:28.936129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:28.936129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:28.936129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:28.939804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 372/747] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:28.940592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:28.955704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:29.053987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f9b279-905b-4062-affc-1e3c6ded2902 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.071354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6b6843-c930-45d3-84f6-ef0ee8c63a33 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.101184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.101792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.102067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.102332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.103017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:29.103318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.104847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.113688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.114685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.115177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.115654 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.118074 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 376/748] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:29.119163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:29.127140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.128904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:29.143717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:29.144742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:29.145659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:29.148547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 373/749] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:41:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:29.149996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:29.171484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:29.212943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:29.235847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1164d6aa-d08f-43a6-b650-cb3c5ffcc1a6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.270287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.270881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.271154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.271435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.272314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:29.272420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.273681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.275156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a4028d-874c-4664-9412-3ba033609f73 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.291163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 377/750] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:29.291163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:29.315771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.316410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.316758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.317107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.317891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=af7a48729497436784b988c6291cfa5d) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:29.318268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.319769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.326028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 374/751] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:29 2022] GET /identity/v3/role_assignments?user.id=af7a48729497436784b988c6291cfa5d&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:29.326902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:29.418414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ecf503-bf2f-4e82-b623-7e75b2227ab9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.443211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:29.462895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.463506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.463783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.464101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.464832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:29.465131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.467033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.474790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78666798-60ef-4ef0-9946-665d49877921 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.479812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:29.498535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 378/752] 10.222.0.42 () {56 vars in 1149 bytes} [Fri Sep 16 01:41:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:29.499767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:29.533751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.534886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.535144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.536028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.536335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.538372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.564174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 375/753] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:41:29 2022] POST /identity/v3/services => generated 240 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:29.565373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:29.663558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ef95af-d6ec-45b4-8507-b5b1f74a5fc7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.717872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.718769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.719284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.719714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.729012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.732324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:29.747057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:29.748725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:29.749692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:29.751915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 379/754] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:41:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:29.757524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:29.775607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54a6c8-c7a1-43e6-bc4d-f7b11d73ecbb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.817773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.818632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.819051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.819485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.829587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:29.834790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:29.894792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12674d55-1d66-4006-9a88-9cf56707276e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:29.905881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.908148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.940985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:29.941501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:29.941852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:29.942045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:29.942766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:29.943063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:29.945314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:29.965094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 380/755] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:41:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:29.972103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.032801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 376/756] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:29 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:41:30.038225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.083358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:30.089123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b49041d-26ea-45e2-b083-271d867f2b98 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:30.103677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:30.125243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:30.125872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:30.126145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:30.126407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:30.136935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:30.139215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:30.151143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f51c83-da6e-4c12-a32e-b10f0d6fafd0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:30.155218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:30.156172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:30.157318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:30.157318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:30.157318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:30.157318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 381/757] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:30.166480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.198554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:30.199101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:30.199377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:30.199643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:30.218149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:30.219600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:30.251125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:30.253512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:30.255554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:30.255554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:30.255554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:30.255554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 377/758] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:30.258976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.368842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23604d6-0b1e-4090-93fc-811b3ac394cb None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:30.421819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:30.430868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0334338-2916-4974-b3cf-6c0fcda9edc9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:30.447277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:30.448363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:30.448929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:30.449402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:30.450300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:30.451135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:30.453108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:30.468237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 382/759] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:30.477739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.480740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:30.482945 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:30.483163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:30.483437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:30.484608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:30.484913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:30.489999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:30.515022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 378/760] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:30.519844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.540164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:30.629996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd55fd1-78f2-4d7e-962f-888ca3915bd2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:30.667968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:30.668599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:30.668876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:30.669961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:30.677503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:30.678944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:30.695720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:30.696680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:30.697583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 379/761] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:30.703732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.718282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:30.803954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ff39d7-b4f5-482c-8b25-694649d0ba22 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:30.814429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7653162-baee-47cc-b07a-4695c18873f0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:30.852791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:30.853396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:30.853672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:30.853940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:30.854699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:30.855005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:30.856640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:30.871139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 380/762] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:30.877733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.940714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:30.941369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:30.941895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:30.941895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:30.969179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:30.970550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:30.982877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:30.983765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:30.984672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 383/763] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 507 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:30.985757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:30.990749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:30.999582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4b9395-f256-4e8a-a61c-330c05251cad None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:31.062319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2db07d-4509-4e17-a63b-e88dfacf0985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:31.062922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2db07d-4509-4e17-a63b-e88dfacf0985 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.063223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2db07d-4509-4e17-a63b-e88dfacf0985 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.063893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2db07d-4509-4e17-a63b-e88dfacf0985 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.064656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2db07d-4509-4e17-a63b-e88dfacf0985 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=f58ec160611e446193b43582a8155284) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:31.067958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2db07d-4509-4e17-a63b-e88dfacf0985 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:31.067958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2db07d-4509-4e17-a63b-e88dfacf0985 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:31.072715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 381/764] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/role_assignments?user.id=f58ec160611e446193b43582a8155284&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:31.080894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1777acb-daa1-440c-a559-995bf678385f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.081243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1777acb-daa1-440c-a559-995bf678385f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.081451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1777acb-daa1-440c-a559-995bf678385f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.085900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 382/765] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:31 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:31.096581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7043769-9d6b-45c1-80b9-58fd61c0dd5c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.096897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7043769-9d6b-45c1-80b9-58fd61c0dd5c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.097124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7043769-9d6b-45c1-80b9-58fd61c0dd5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.130214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d7043769-9d6b-45c1-80b9-58fd61c0dd5c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:31.132735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d7043769-9d6b-45c1-80b9-58fd61c0dd5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:31.168239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:31.206282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089dfec4-661f-46cc-86f6-5eb25bd28ac0 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:31.280202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 383/766] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 364 bytes (3 switches on core 0) Sep 16 01:41:31.288374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:31.289000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.289276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.289542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.290235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:31.290537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:31.293196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:31.310976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 384/767] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:31.316859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7043769-9d6b-45c1-80b9-58fd61c0dd5c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:31.404145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:31.451156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7043769-9d6b-45c1-80b9-58fd61c0dd5c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:31.487532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:31.488208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.488534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.488807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/603c81d781754201a5ac5b7d9b84fab6/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.490686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=603c81d781754201a5ac5b7d9b84fab6, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:31.492140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:31.506940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 384/768] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:31 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/603c81d781754201a5ac5b7d9b84fab6/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:31.682724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:31.808808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fba3be-5075-45f6-8c46-7aa6e51366b9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:31.847124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9758e77-a929-41aa-9bf2-e7a7239181f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:31.847751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9758e77-a929-41aa-9bf2-e7a7239181f7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.848087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9758e77-a929-41aa-9bf2-e7a7239181f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.848354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9758e77-a929-41aa-9bf2-e7a7239181f7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.848935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9758e77-a929-41aa-9bf2-e7a7239181f7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:31.850292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9758e77-a929-41aa-9bf2-e7a7239181f7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-c9758e77-a929-41aa-9bf2-e7a7239181f7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:31.859558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:31.860624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:31.861456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 385/769] 10.222.0.42 () {56 vars in 1129 bytes} [Fri Sep 16 01:41:31 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:31.867025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:31.878943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d9a5b9-b920-4d9f-85ff-52d4c335c845 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.879243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d9a5b9-b920-4d9f-85ff-52d4c335c845 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.879474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d9a5b9-b920-4d9f-85ff-52d4c335c845 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.881088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 386/770] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:31.889051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920baf28-5e8e-46af-9350-56cfa678fbb5 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:31.889394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920baf28-5e8e-46af-9350-56cfa678fbb5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:31.889575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920baf28-5e8e-46af-9350-56cfa678fbb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:31.924952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-920baf28-5e8e-46af-9350-56cfa678fbb5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:31.927263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-920baf28-5e8e-46af-9350-56cfa678fbb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:31.936080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:32.022188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2abe8d-b6b5-4253-8ecf-aa353186f49a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:32.048399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:32.069244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:32.070588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.071415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.072435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.073896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 387/771] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:31 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:32.075282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:32.076604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:32.078247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:32.083727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 385/772] 10.222.0.42 () {56 vars in 1124 bytes} [Fri Sep 16 01:41:31 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:32.088999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920baf28-5e8e-46af-9350-56cfa678fbb5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:32.226752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920baf28-5e8e-46af-9350-56cfa678fbb5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:32.284621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3985de21-b915-46d4-b7be-d301cbe6721c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:32.285727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3985de21-b915-46d4-b7be-d301cbe6721c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.286208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3985de21-b915-46d4-b7be-d301cbe6721c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.286676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3985de21-b915-46d4-b7be-d301cbe6721c None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.287504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3985de21-b915-46d4-b7be-d301cbe6721c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:32.288043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3985de21-b915-46d4-b7be-d301cbe6721c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:32.289599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3985de21-b915-46d4-b7be-d301cbe6721c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:32.300392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 388/773] 10.222.0.42 () {56 vars in 1124 bytes} [Fri Sep 16 01:41:32 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:32.305683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:32.338368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be7580-2dae-42b9-ae62-5528558dae21 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.338368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be7580-2dae-42b9-ae62-5528558dae21 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.338368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12be7580-2dae-42b9-ae62-5528558dae21 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.339226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 389/774] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:32.358115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcba231-90d9-4cea-8b8f-26c41f9828e7 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.358398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcba231-90d9-4cea-8b8f-26c41f9828e7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.358652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcba231-90d9-4cea-8b8f-26c41f9828e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.376607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:32.381905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8bcba231-90d9-4cea-8b8f-26c41f9828e7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:32.383087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8bcba231-90d9-4cea-8b8f-26c41f9828e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:32.446075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a024b5-dc32-4e17-9b3f-81719887af72 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:32.458303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 390/775] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:32 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:32.459256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcba231-90d9-4cea-8b8f-26c41f9828e7 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:32.512661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:32.513418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.513873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.514142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.514839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] RBAC: Adding query filter params (interface=public, service_id=84fe613f21294c9da9c345e54bec8ab6, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:32.515139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:32.518526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:32.533614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 386/776] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:41:32 2022] GET /identity/v3/endpoints?service_id=84fe613f21294c9da9c345e54bec8ab6&interface=public®ion_id=RegionOne => generated 194 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:32.539219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:32.595827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcba231-90d9-4cea-8b8f-26c41f9828e7 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:32.636548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:32.643661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:32.646147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:32.646793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.647076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.647351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.661708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:32.666756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:32.667857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:32.668906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 391/777] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:32.674019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:32.738857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23192511-36a6-43a9-a5dc-25fcce1d2540 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:32.802579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:32.803109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.803384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.803651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.804309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:32.807238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:32.812492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 387/778] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:41:32 2022] GET /identity/v3/services => generated 772 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:32.813773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:32.852563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e27e01-b6b7-490c-979e-63f16f951a2d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:32.891842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:32.893248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:32.893946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:32.894675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:32.896143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:32.896674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:32.898416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:32.909864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 392/779] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:32.916024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:32.996819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a286820-855b-4135-9083-beffe1c888c4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.018441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:33.039720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:33.076488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.077182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.077453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.077719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] PATH_INFO: `/v3/services/network` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.078356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.079610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3745a0-1573-4d72-b927-c74e2ce05dd5 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.080067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:33.089183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:33.090204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:33.091284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 388/780] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:32 2022] GET /identity/v3/services/network => generated 88 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:33.107064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:33.126857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.127571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.127917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.128281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.141505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.143047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:33.159165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:33.160181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:33.161067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:33.161067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:33.161067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:33.161067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 393/781] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:33.166415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:33.242833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa9bcf-5f72-4354-8d44-0a3de1f88114 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.286083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.286798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.287228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.287445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.288182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] RBAC: Adding query filter params (name=network) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:33.288547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.290372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.294351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5465d350-19bd-4a79-9030-b56eb6326216 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.295742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 389/782] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:33 2022] GET /identity/v3/services?name=network => generated 124 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:33.302519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:33.304105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:33.369486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:33.373085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.373774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.373988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.374302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.374958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:33.375258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.376665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.400969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 394/783] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:33.407204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:33.439061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390ab0d9-2b86-4d2f-b91f-f49100023ee2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.480000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.481300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.481930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.482386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.483156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] RBAC: Adding query filter params (type=network) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:33.483689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.485634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.491498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 390/784] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:33 2022] GET /identity/v3/services?type=network => generated 350 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:33.498012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:33.609622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e1f32-47fd-460e-b29f-59dc564406cc None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.657166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.658267 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.658698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.659114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] PATH_INFO: `/v3/users/swift` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.668304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.670600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:33.683868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:33.685232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:33.686596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:33.689337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 395/785] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:33.695373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:33.730891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:33.770598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8221c557-85e4-42dd-b091-02c8e0a8081e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.780817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:33.815828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a35bc12-87c0-452a-b0c9-c503cfe93687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.816809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35bc12-87c0-452a-b0c9-c503cfe93687 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.817238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35bc12-87c0-452a-b0c9-c503cfe93687 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.817724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35bc12-87c0-452a-b0c9-c503cfe93687 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.818632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a35bc12-87c0-452a-b0c9-c503cfe93687 None admin] RBAC: Adding query filter params (interface=public, service_id=23b6637def624574bbb87b63bc94941b, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:33.819087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a35bc12-87c0-452a-b0c9-c503cfe93687 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.820791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a35bc12-87c0-452a-b0c9-c503cfe93687 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.831989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9886e980-c237-4d4e-a9ca-552f0b67accd None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.832307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 391/786] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:41:33 2022] GET /identity/v3/endpoints?service_id=23b6637def624574bbb87b63bc94941b&interface=public®ion_id=RegionOne => generated 194 bytes in 332 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:33.833638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57f123d-1462-48c5-ad12-f40add61f50e None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.833923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57f123d-1462-48c5-ad12-f40add61f50e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.834195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57f123d-1462-48c5-ad12-f40add61f50e None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.839977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 392/787] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:33 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:33.840787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57f123d-1462-48c5-ad12-f40add61f50e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:33.872230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86eb2d6-ac26-47f3-82f7-c901ef2e2701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.872857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86eb2d6-ac26-47f3-82f7-c901ef2e2701 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86eb2d6-ac26-47f3-82f7-c901ef2e2701 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.873516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86eb2d6-ac26-47f3-82f7-c901ef2e2701 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.874355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86eb2d6-ac26-47f3-82f7-c901ef2e2701 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:33.874668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86eb2d6-ac26-47f3-82f7-c901ef2e2701 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.876499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86eb2d6-ac26-47f3-82f7-c901ef2e2701 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:33.899212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 396/788] 10.222.0.42 () {56 vars in 1135 bytes} [Fri Sep 16 01:41:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:33.901362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a718c49b-c51f-43df-9f11-6e26e090696b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.901658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a718c49b-c51f-43df-9f11-6e26e090696b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.901939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a718c49b-c51f-43df-9f11-6e26e090696b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.924367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a718c49b-c51f-43df-9f11-6e26e090696b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:33.925587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a718c49b-c51f-43df-9f11-6e26e090696b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:33.950920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57f123d-1462-48c5-ad12-f40add61f50e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:33.987863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:33.988575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:33.988854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:33.989174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:33.989802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:33.991308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:34.004203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 393/789] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:41:33 2022] GET /identity/v3/services => generated 772 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:34.005323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:34.078190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:34.172214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 397/790] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:33 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:34.208516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f37f6d-cc23-46c2-98da-169a8d9adaff None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:34.265648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009585e2-635d-457b-b20a-1a7a37c3e700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:34.266283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009585e2-635d-457b-b20a-1a7a37c3e700 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.266562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009585e2-635d-457b-b20a-1a7a37c3e700 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.266831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009585e2-635d-457b-b20a-1a7a37c3e700 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.286229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009585e2-635d-457b-b20a-1a7a37c3e700 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:34.287700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009585e2-635d-457b-b20a-1a7a37c3e700 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-009585e2-635d-457b-b20a-1a7a37c3e700 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:34.307658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:34.308679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:34.310070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:34.310070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:34.310070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:34.310070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 394/791] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:34.315638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a718c49b-c51f-43df-9f11-6e26e090696b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:34.333648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:34.352745 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c16d65-9d35-4725-84c4-52399568421c None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.353055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c16d65-9d35-4725-84c4-52399568421c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.353290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c16d65-9d35-4725-84c4-52399568421c None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.354623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 395/792] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:34.361218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2d18eb-9fb5-42ca-92f3-fedfa8f3c2df None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.361500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2d18eb-9fb5-42ca-92f3-fedfa8f3c2df None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.361769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2d18eb-9fb5-42ca-92f3-fedfa8f3c2df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.388286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-cf2d18eb-9fb5-42ca-92f3-fedfa8f3c2df None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:34.390170 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cf2d18eb-9fb5-42ca-92f3-fedfa8f3c2df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:34.444464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a718c49b-c51f-43df-9f11-6e26e090696b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:34.478620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 396/793] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:34 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:34.479588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2d18eb-9fb5-42ca-92f3-fedfa8f3c2df None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:34.489340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:34.490377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.490658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.490928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.491577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:34.491877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:34.493680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:34.501384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:34.509469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 398/794] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:34.516167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:34.616772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2d18eb-9fb5-42ca-92f3-fedfa8f3c2df None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:34.644027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:34.680953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:34.681594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.681872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.682139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.682684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e804cde5-b87f-444c-add6-59a211eb52e3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:34.710876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:34.715747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:34.717121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:34.717953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 397/795] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:34.723545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:34.724914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:34.725555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.725833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.726102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.735365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:34.737728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:34.751092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:34.752043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:34.752946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:34.753362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 399/796] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:34.759503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:34.869685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:34.897171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e953435-e58f-4db4-9dca-d636d396b4e2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:34.902592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856fba0f-4ffb-4315-a55f-d1aa588554ea None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:34.943618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:34.944339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.944607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:34.944748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.944908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.945211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:34.945490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:34.945587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:34.945764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:34.945854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:34.946412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:34.946710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:34.947255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:34.948102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:34.949141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:34.961300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 398/797] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:34.962364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:34.969766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 400/798] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:34.970732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.094807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed0f79d-4c85-4a73-8020-908a87ed2439 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.095664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ead0f45-70e1-48ab-bdfd-ff0399f5371d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.139360 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.140124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.140409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.140683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.141814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.142942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.143852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.144367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.152653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:35.154709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:35.155925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:35.160078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:35.165929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:35.166879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:35.167806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:35.171219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 399/799] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:35.171991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:35.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:35.179233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:35.180140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:35.180140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:35.180140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:35.182213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 401/800] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:35.183322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.232949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:35.289525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a151f64b-0d5a-4c91-83e2-8b298adc7d05 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.296942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f1c45-60fc-446c-8f24-f0d102d7ab36 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.328493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.329150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.329423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.329694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/af7a48729497436784b988c6291cfa5d/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.331819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=af7a48729497436784b988c6291cfa5d, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:35.333742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:35.337949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.338934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.339121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.339491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.340389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:35.340724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:35.342661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:35.350796 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 400/801] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:35 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/af7a48729497436784b988c6291cfa5d/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:35.352195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 402/802] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:35.352498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.365957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.432354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:35.474381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e731d95-cca9-4d3b-a4ba-b1d33d750c70 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.485070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e532730-7318-48e7-a65a-d53f8064013c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.496164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:35.517776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf419aa5-a062-411d-973d-082f4c831ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.518857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf419aa5-a062-411d-973d-082f4c831ace None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.519330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf419aa5-a062-411d-973d-082f4c831ace None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.519813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf419aa5-a062-411d-973d-082f4c831ace None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.520829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf419aa5-a062-411d-973d-082f4c831ace None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:35.521423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf419aa5-a062-411d-973d-082f4c831ace None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:35.523689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf419aa5-a062-411d-973d-082f4c831ace None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:35.526931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44d050c-4858-4e3b-8598-0ebebde93c5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.527575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d050c-4858-4e3b-8598-0ebebde93c5d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.527905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d050c-4858-4e3b-8598-0ebebde93c5d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.528200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44d050c-4858-4e3b-8598-0ebebde93c5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.543095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d050c-4858-4e3b-8598-0ebebde93c5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:35.543095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44d050c-4858-4e3b-8598-0ebebde93c5d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:35.543696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 401/803] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:35.545455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9353b7-5a0e-45dc-ae9f-2732c8f6986b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.545948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9353b7-5a0e-45dc-ae9f-2732c8f6986b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.546388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9353b7-5a0e-45dc-ae9f-2732c8f6986b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.549073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 402/804] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:35.551210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9353b7-5a0e-45dc-ae9f-2732c8f6986b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-b44d050c-4858-4e3b-8598-0ebebde93c5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:35.557989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:35.558915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:35.560435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:35.560435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:35.560435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:35.560435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 403/805] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:35.562362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd8b06-6be7-4f8a-8ee5-d63aac2015b8 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.562362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd8b06-6be7-4f8a-8ee5-d63aac2015b8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.562362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dd8b06-6be7-4f8a-8ee5-d63aac2015b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.582415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-16dd8b06-6be7-4f8a-8ee5-d63aac2015b8 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:35.583608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-16dd8b06-6be7-4f8a-8ee5-d63aac2015b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:35.672483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 404/806] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:35 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:35.673502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd8b06-6be7-4f8a-8ee5-d63aac2015b8 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.674896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9353b7-5a0e-45dc-ae9f-2732c8f6986b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.715541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:35.758822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.759466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.759744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.760045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.768447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:35.769901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:35.782965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:35.783960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:35.784882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 403/807] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:41:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:35.792217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:35.802212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.815982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dd8b06-6be7-4f8a-8ee5-d63aac2015b8 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:35.860404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:35.861683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:35.862951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:35.863446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:35.865389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:35.867101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:35.868871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:35.895767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 405/808] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:35.901378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:35.989684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37684724-079d-41cf-8509-77e5692867d0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.038208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.038208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.038208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.038208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.038670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:36.038670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.038670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.053160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88315ecd-82c1-41ce-97b7-eba1689eb0d6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.059432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 404/809] 10.222.0.42 () {56 vars in 1149 bytes} [Fri Sep 16 01:41:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:36.060993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.098214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.099179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.099635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.100107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] PATH_INFO: `/v3/users/nova` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.109432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.111141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.112950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:36.125516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:36.126565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:41:36.127495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:36.129721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 406/810] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:41:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:36.130727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.145162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:36.197384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d53e77-da23-4192-b0ee-7b3ce55f9e7e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.243213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.244232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.244712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.245200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.245987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.248149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.254892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5c5137-bcf2-4de7-b1ca-0e59f4f9d2ee None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:36.258276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:36.259256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 16 01:41:36.260111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:36.268598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 405/811] 10.222.0.42 () {56 vars in 1129 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:36.269721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.296903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.297683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.298090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.298378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.316671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.318588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:36.342726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:36.343747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:36.344664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:36.344664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:36.344664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:36.344664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 407/812] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:36.345381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.395620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fba0c8-e65a-4470-a68f-64e6df6ae4dc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.397338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:36.441166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:36.441981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.442657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.442938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.443287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.443973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:36.444599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.446472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.465294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a288c995-3c89-4295-a84d-d651ede9c32b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.470006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 406/813] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:36.470936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.504172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.505085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.505582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.506047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.506919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:36.507412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.509085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.519116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 408/814] 10.222.0.42 () {56 vars in 1124 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:36.520351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.587195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c22ac4-9ce2-4251-8561-50941035a785 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.624831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.625654 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.625938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.626211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.626875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:36.627572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.629191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.646487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa6823f-e5d9-4518-bbc1-4ae52209876e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.647579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 407/815] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:36.648621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.679196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:36.682687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:36.694229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.695878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.696508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.696956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.783501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.784952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.895350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d78ba44-eaaf-4cbb-aaf9-f28452c577b1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:36.904298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 409/816] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:41:36.904298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:36.931591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a67a495-e3f7-4de5-90ff-b46a932669e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:36.932541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67a495-e3f7-4de5-90ff-b46a932669e2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.932976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67a495-e3f7-4de5-90ff-b46a932669e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.933448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a67a495-e3f7-4de5-90ff-b46a932669e2 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.934065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a67a495-e3f7-4de5-90ff-b46a932669e2 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:36.934363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a67a495-e3f7-4de5-90ff-b46a932669e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:36.934505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:36.935876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a67a495-e3f7-4de5-90ff-b46a932669e2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:36.945430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 408/817] 10.222.0.42 () {56 vars in 1124 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:36.945430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f2658-be0b-4c96-acf5-90ec0c827169 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:36.945430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f2658-be0b-4c96-acf5-90ec0c827169 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:36.945430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f2658-be0b-4c96-acf5-90ec0c827169 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:36.945430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 409/818] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:36 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:36.945430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2f2658-be0b-4c96-acf5-90ec0c827169 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.021793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f90475d-fa58-4f50-b1ca-63087870ffa4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.036559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:37.055176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2f2658-be0b-4c96-acf5-90ec0c827169 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.061646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.062275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.062552 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.062818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.071219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.072662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:37.084497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:37.085555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:37.086802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:37.090325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 410/819] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:37.091518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.095209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf92ef0-706b-41aa-93b5-8e2a91e68be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.095837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf92ef0-706b-41aa-93b5-8e2a91e68be7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.096142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf92ef0-706b-41aa-93b5-8e2a91e68be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.096407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf92ef0-706b-41aa-93b5-8e2a91e68be7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.107637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf92ef0-706b-41aa-93b5-8e2a91e68be7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.109162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf92ef0-706b-41aa-93b5-8e2a91e68be7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-3cf92ef0-706b-41aa-93b5-8e2a91e68be7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:37.124032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:37.125533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:37.126449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:37.126449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:37.126449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:37.126449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 410/820] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:37.126449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc83b625-32a8-4281-a7e4-244613258719 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.126764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc83b625-32a8-4281-a7e4-244613258719 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.127566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc83b625-32a8-4281-a7e4-244613258719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.145825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bc83b625-32a8-4281-a7e4-244613258719 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:37.147008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bc83b625-32a8-4281-a7e4-244613258719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:37.192654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:37.208225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba82b5f-7255-4fc5-9a80-dbf02d6800d5 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.220355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 411/821] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:37.221437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc83b625-32a8-4281-a7e4-244613258719 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.251967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.252808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.253153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.253498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.254260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.255791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.288232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 411/822] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:41:37 2022] POST /identity/v3/endpoints => generated 346 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:37.291427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.333399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:37.396364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc83b625-32a8-4281-a7e4-244613258719 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.476034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.478220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.480228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.480733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.482216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:37.482735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.484403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.496857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1701615-c982-4931-92fb-eb9f540810a1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.501783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 412/823] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:37.510056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.550961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.551970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.552476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.552935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.553681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:37.554550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.555389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.569407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 412/824] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:37.570216 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.573202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:37.654488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ba995d-98ed-4577-b62f-b13d1d2a2a2e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.681596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:37.691218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3cee38-474d-47f8-aeef-6199d988467f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.694382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.695032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.695309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.695574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/f58ec160611e446193b43582a8155284/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.711624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=f58ec160611e446193b43582a8155284, role_id=cb16002d0a2b4241b1d9c4c8ecd9ba42)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.713104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.727144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 413/825] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:37 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/f58ec160611e446193b43582a8155284/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:37.727963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.738319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e9715c-c80a-4232-a6a4-da8a76134f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.739393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9715c-c80a-4232-a6a4-da8a76134f6b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.739869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9715c-c80a-4232-a6a4-da8a76134f6b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.740430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9715c-c80a-4232-a6a4-da8a76134f6b None admin] PATH_INFO: `/v3/services/network` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.741386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e9715c-c80a-4232-a6a4-da8a76134f6b None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.743089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e9715c-c80a-4232-a6a4-da8a76134f6b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-96e9715c-c80a-4232-a6a4-da8a76134f6b None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:37.755317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:37.756398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 16 01:41:37.757271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:37.760823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 413/826] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:37 2022] GET /identity/v3/services/network => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:37.763109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad48cc9b-b8dd-426d-88f7-866ff4de57b0 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.763402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad48cc9b-b8dd-426d-88f7-866ff4de57b0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.763753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad48cc9b-b8dd-426d-88f7-866ff4de57b0 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.765449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 414/827] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:37.766919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad48cc9b-b8dd-426d-88f7-866ff4de57b0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.863974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:37.866843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2e7715-58ca-4ab3-a95a-320ab2786462 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.909605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d415cd-d8ea-4e66-ac41-0fe796aa2671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.910632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d415cd-d8ea-4e66-ac41-0fe796aa2671 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.911132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d415cd-d8ea-4e66-ac41-0fe796aa2671 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.911612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d415cd-d8ea-4e66-ac41-0fe796aa2671 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.913279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad48cc9b-b8dd-426d-88f7-866ff4de57b0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:37.920455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d415cd-d8ea-4e66-ac41-0fe796aa2671 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.922267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d415cd-d8ea-4e66-ac41-0fe796aa2671 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a2d415cd-d8ea-4e66-ac41-0fe796aa2671 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:37.937069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:37.938054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:37.938925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 414/828] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:37.943981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad14d4-2c3d-4bbf-bdd2-7aa6e649838f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.943981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad14d4-2c3d-4bbf-bdd2-7aa6e649838f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.943981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad14d4-2c3d-4bbf-bdd2-7aa6e649838f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.959734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:37.960368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:37.960643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:37.960906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:37.961523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] RBAC: Adding query filter params (name=network) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:37.961817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:37.962749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-83ad14d4-2c3d-4bbf-bdd2-7aa6e649838f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:37.963148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:37.964024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-83ad14d4-2c3d-4bbf-bdd2-7aa6e649838f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:37.968719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 415/829] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:37 2022] GET /identity/v3/services?name=network => generated 124 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:37.969747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:37.997919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:38.045322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 415/830] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:37 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:38.046376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ad14d4-2c3d-4bbf-bdd2-7aa6e649838f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:38.084433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf1d403-8cd7-440b-b23c-77ae00678ef8 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:38.114373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:38.121170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:38.122758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:38.123403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:38.123808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:38.124697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:38.125127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:38.126670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:38.140936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 416/831] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:38.146650 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:38.189979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ad14d4-2c3d-4bbf-bdd2-7aa6e649838f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:38.253275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:38.254129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:38.254828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:38.255241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:38.256300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] RBAC: Adding query filter params (type=network) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:38.256816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:38.258434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:38.264268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 416/832] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:38 2022] GET /identity/v3/services?type=network => generated 350 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:38.269477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:38.328320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:38.356201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7764296-8de1-41fe-91e3-a0726e88e3ad None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:38.412075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:38.412955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:38.413231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:38.413496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:38.414466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=603c81d781754201a5ac5b7d9b84fab6) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:38.414769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:38.417013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:38.433370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 417/833] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:38 2022] GET /identity/v3/role_assignments?user.id=603c81d781754201a5ac5b7d9b84fab6&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:38.434354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:38.453477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7717468a-2729-4899-8b24-123a8b956ecc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:38.534601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:38.535338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:38.535612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:38.536186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:38.536881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:38.538258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:38.561660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:38.582467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 417/834] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:41:38 2022] POST /identity/v3/endpoints => generated 337 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:38.632653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84536deb-2af1-4754-8064-222de55ee8f6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:38.679037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:38.679748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:38.680069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:38.680338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:38.693468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:38.699103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:38.700199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:41:38.701411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:38.705548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 418/835] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:38.707063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:38.808084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:38.828826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915b789f-4ce4-42bc-bf55-265bb8c2700b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:38.866122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:38.866724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:38.867001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:38.867262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:38.867916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:38.868252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:38.869629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:38.882255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 418/836] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:38.888749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:39.004537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6a9ac-cdd7-4bb3-810a-fa1dc75f07a5 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:39.016299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:39.045318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:39.045945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:39.046218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:39.046577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:39.058803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:39.060669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:39.078180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:39.079149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:39.080053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:39.080053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:39.080053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:39.084047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 419/837] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:39.089143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:39.194396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559382c-d740-45b6-a3c6-39562011b401 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:39.229981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:39.230588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:39.230858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:39.231122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:39.232142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:39.232472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:39.233829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:39.250951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 419/838] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:39.252389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:39.355086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74034a60-61af-4293-bffa-edf9d97f243d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:39.391247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:39.391868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:39.392171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:39.392445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:39.401370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:39.402799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:39.415180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:39.416122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:39.417051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 420/839] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:41:39 2022] GET /identity/v3/users/swift => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:39.422790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:39.426290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:39.530059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b07de3-59cb-4fb7-9996-3d4d1e1fb369 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:39.565916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adacd4f-33ec-4890-a0d5-c6b8f041c468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:39.566906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adacd4f-33ec-4890-a0d5-c6b8f041c468 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:39.567384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adacd4f-33ec-4890-a0d5-c6b8f041c468 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:39.567962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adacd4f-33ec-4890-a0d5-c6b8f041c468 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:39.568883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adacd4f-33ec-4890-a0d5-c6b8f041c468 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:39.569384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adacd4f-33ec-4890-a0d5-c6b8f041c468 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:39.571003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adacd4f-33ec-4890-a0d5-c6b8f041c468 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:39.589420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 420/840] 10.222.0.42 () {56 vars in 1135 bytes} [Fri Sep 16 01:41:39 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:39.596266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:39.624634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:39.706322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6251514f-e02c-4c2e-be53-aa370d3d9935 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:39.727014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6597d54-26aa-498e-a89f-5d7d89070028 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:39.727303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6597d54-26aa-498e-a89f-5d7d89070028 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:39.727565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6597d54-26aa-498e-a89f-5d7d89070028 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:39.729158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 421/841] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:39.734563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fddc43-2f33-41b5-97d3-d7fe3b16da1f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:39.734840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fddc43-2f33-41b5-97d3-d7fe3b16da1f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:39.735108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fddc43-2f33-41b5-97d3-d7fe3b16da1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:39.741595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:39.742193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:39.742466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:39.742719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:39.753260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d3fddc43-2f33-41b5-97d3-d7fe3b16da1f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:39.754717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d3fddc43-2f33-41b5-97d3-d7fe3b16da1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:39.818151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:39.823015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:39.826094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:39.839961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 422/842] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:39 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:39.859026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:39.944070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 421/843] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:39 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:41:39.946130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fddc43-2f33-41b5-97d3-d7fe3b16da1f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.050948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fddc43-2f33-41b5-97d3-d7fe3b16da1f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.086562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.087122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.087401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.087670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.100350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.101831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:40.119054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:40.120136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:40.121020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:40.121020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:40.121020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:40.121020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 423/844] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:40.127317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.128970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.179817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:40.234025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4beccf2-dbcb-4ff6-b8c8-934150452956 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.234968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44af86d1-de1b-4aa1-999b-a5b04f14da3d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.260237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:40.271190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e1775b-fb69-429a-b60d-a4eb079b71a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.271812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e1775b-fb69-429a-b60d-a4eb079b71a5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.272215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e1775b-fb69-429a-b60d-a4eb079b71a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.272584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e1775b-fb69-429a-b60d-a4eb079b71a5 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.273161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e1775b-fb69-429a-b60d-a4eb079b71a5 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.273504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc511168-e6e3-4a3c-baf6-221577ba367b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.274160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc511168-e6e3-4a3c-baf6-221577ba367b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.274443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc511168-e6e3-4a3c-baf6-221577ba367b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.274537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e1775b-fb69-429a-b60d-a4eb079b71a5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.274708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc511168-e6e3-4a3c-baf6-221577ba367b None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.275339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc511168-e6e3-4a3c-baf6-221577ba367b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:40.275635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc511168-e6e3-4a3c-baf6-221577ba367b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.277481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc511168-e6e3-4a3c-baf6-221577ba367b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-32e1775b-fb69-429a-b60d-a4eb079b71a5 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:40.283162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:40.284182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:40.290524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 424/845] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:40.291329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76c8d4-4c75-459e-96cf-d50c200243fb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.291329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76c8d4-4c75-459e-96cf-d50c200243fb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.291329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d76c8d4-4c75-459e-96cf-d50c200243fb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.291329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 425/846] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:40.291329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76c8d4-4c75-459e-96cf-d50c200243fb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.296926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 422/847] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:40.301942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afabd053-bab8-4a21-a491-713a220cce3e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.301942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afabd053-bab8-4a21-a491-713a220cce3e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.301942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afabd053-bab8-4a21-a491-713a220cce3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.318063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-afabd053-bab8-4a21-a491-713a220cce3e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:40.319325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-afabd053-bab8-4a21-a491-713a220cce3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:40.395862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 423/848] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:40 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:40.396747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afabd053-bab8-4a21-a491-713a220cce3e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.409162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d76c8d4-4c75-459e-96cf-d50c200243fb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.422157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:40.456264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.456865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.457134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.457393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.458014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:40.458308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.460660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.472867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 426/849] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:40.474267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.525349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afabd053-bab8-4a21-a491-713a220cce3e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.540083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:40.565506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.566191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.566470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.566736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.574433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.576523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:40.589781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:40.590763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:40.591646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:40.596161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 424/850] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:40.597488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.602858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69105fc5-a0da-489c-9d5d-382b07dbd916 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.649700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.650922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.651407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.652138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.653703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:40.654499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.656731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.662464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 427/851] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:40.664080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.716499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5547ecd-4e83-4262-a30a-4809879fde13 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.717418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:40.762153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.763307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.763907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.764398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.765400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:40.765920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.767784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.789436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 425/852] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:40.791195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.809708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2af15ff-30b1-4fd7-8665-87590acb08c5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.830878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:40.858552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.859533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.859533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.859824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.872813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:40.878374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:40.879372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:40.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 428/853] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:40.886552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:40.920530 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c102783-e10c-41e9-a1b9-a3d8b275f6d1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:40.959684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-025299ff-a9ac-44a9-92e0-c805ee9dc9d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:40.960346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025299ff-a9ac-44a9-92e0-c805ee9dc9d1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:40.960621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025299ff-a9ac-44a9-92e0-c805ee9dc9d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:40.960891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-025299ff-a9ac-44a9-92e0-c805ee9dc9d1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:40.961780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025299ff-a9ac-44a9-92e0-c805ee9dc9d1 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=af7a48729497436784b988c6291cfa5d) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:40.962469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025299ff-a9ac-44a9-92e0-c805ee9dc9d1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:40.964211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-025299ff-a9ac-44a9-92e0-c805ee9dc9d1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:40.972295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 426/854] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/role_assignments?user.id=af7a48729497436784b988c6291cfa5d&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:40.999763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bc7a16-0437-4308-ae55-a263071b3ecc None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.000115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bc7a16-0437-4308-ae55-a263071b3ecc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.000356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bc7a16-0437-4308-ae55-a263071b3ecc None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.002004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 427/855] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:41.008047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378ecb6-33bd-4b30-abd5-13c515fde169 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.008390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378ecb6-33bd-4b30-abd5-13c515fde169 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.008687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a378ecb6-33bd-4b30-abd5-13c515fde169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.014023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:41.021464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51282955-22c2-41d5-994d-12f8a32bbbb2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.037370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a378ecb6-33bd-4b30-abd5-13c515fde169 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:41.039235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a378ecb6-33bd-4b30-abd5-13c515fde169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:41.061777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.062556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.062988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.063375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.064261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:41.064672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.066153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.076281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 429/856] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:41:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:41.081982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.141333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 428/857] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:41 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:41.202561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bb5c7-1c28-48ed-af97-da223a3f7eb5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.239033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.240483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.240963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.241485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.252951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.254827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:41.270721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:41.271726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:41.272649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:41.272649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:41.272649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:41.278633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 430/858] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:41.281507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a378ecb6-33bd-4b30-abd5-13c515fde169 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.311258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:41.389389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a378ecb6-33bd-4b30-abd5-13c515fde169 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.412604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.426049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.426607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:41.426694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.426922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.427183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.427831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:41.428159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.429684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.446898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 429/859] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:41.448859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.548661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:41.549964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dc2f93-9094-4672-b59d-56e84eced71e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.567903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c588b8fb-916a-430c-b7a1-3d1d603d4c99 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.596552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.597190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.597517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.597793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.598381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.600194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:41.610033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:41.611033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:41.612743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 431/860] 10.222.0.42 () {56 vars in 1132 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:41.613784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.615827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.615827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.615827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.617139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.627351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.628838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:41.643017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:41.644023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:41.645048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 430/861] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:41.649922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.725933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:41.756807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-441d3ca1-3a6c-44f9-b955-cdd3fcc40e27 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.765481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b3dec3-1e78-44bf-9aed-088b87ac82f6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.802696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.803227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.803497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.803756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.809788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:41.810109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.811916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.813686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.814579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.815050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.815518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.816494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:41.816975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.817246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 432/862] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:41.818921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.827965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.844717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 431/863] 10.222.0.42 () {56 vars in 1149 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:41.846106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:41.849953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:41.931466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2867566c-d5aa-42f6-82b8-88b24c824b6f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.955464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b39478-cfd6-4f3c-8b97-cf6e46ad765f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:41.968387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23e713-202c-4203-98bc-098e813640ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.969203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23e713-202c-4203-98bc-098e813640ff None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.969654 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23e713-202c-4203-98bc-098e813640ff None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.970064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23e713-202c-4203-98bc-098e813640ff None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:41.970936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af23e713-202c-4203-98bc-098e813640ff None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:41.971477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af23e713-202c-4203-98bc-098e813640ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:41.973010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af23e713-202c-4203-98bc-098e813640ff None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:41.983650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 433/864] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:41.994532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:41.995173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:41.995453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:41.995722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:42.074589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:42.076615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:42.079715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:42.186000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 432/865] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:41 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 16 01:41:42.191705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23e713-202c-4203-98bc-098e813640ff None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:42.194608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:42.302560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23e713-202c-4203-98bc-098e813640ff None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:42.338854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:42.339403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:42.339679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:42.339967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:42.351860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:42.353323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:42.368292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:42.369285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:42.370466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:42.370466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:42.370466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:42.370466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 434/866] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:42.377100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:42.489848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfd166f-e62d-498b-b187-98b3266b6276 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:42.493047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:42.527873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:42.529389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:42.529902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:42.530386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:42.531238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:42.531737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:42.533306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:42.549960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 433/867] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:42.551541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:42.610207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:42.657729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6c7a6a-1440-4038-87ad-c25031012552 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:42.692850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:42.693473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:42.693749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:42.694015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:42.701732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:42.703170 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:42.714550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:42.715522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:42.716682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 435/868] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:42.723266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:42.838144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddfa613-0da9-4ad5-89cd-9644cff58256 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:42.876268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9ce477-332a-4cb4-ac8c-a7d89642fbc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:42.876793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9ce477-332a-4cb4-ac8c-a7d89642fbc9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:42.877072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9ce477-332a-4cb4-ac8c-a7d89642fbc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:42.877345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9ce477-332a-4cb4-ac8c-a7d89642fbc9 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:42.878030 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9ce477-332a-4cb4-ac8c-a7d89642fbc9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:42.878330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9ce477-332a-4cb4-ac8c-a7d89642fbc9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:42.879733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9ce477-332a-4cb4-ac8c-a7d89642fbc9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:42.898026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 434/869] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:42.899414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:42.957215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b0af-3352-4818-bb58-cb7c75cdaec7 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:42.957871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b0af-3352-4818-bb58-cb7c75cdaec7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:42.958368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b0af-3352-4818-bb58-cb7c75cdaec7 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:42.960196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 435/870] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:42.965732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdd506-6970-4541-b55b-acbf87b44b2e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:42.966528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdd506-6970-4541-b55b-acbf87b44b2e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:42.966997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bdd506-6970-4541-b55b-acbf87b44b2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:42.986352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a1bdd506-6970-4541-b55b-acbf87b44b2e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:42.987866 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a1bdd506-6970-4541-b55b-acbf87b44b2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:43.006786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:43.019345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9beba7dd-3a43-4c64-b87e-6532e212f2d7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:43.037337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:43.067961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:43.067961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:43.067961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:43.067961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:43.068380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=f58ec160611e446193b43582a8155284) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:43.068380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:43.068745 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:43.069169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 436/871] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:42 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:43.080028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 436/872] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:42 2022] GET /identity/v3/role_assignments?user.id=f58ec160611e446193b43582a8155284&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:43.333758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b409a96-8219-4fe5-9207-b459e891a3f8 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:43.345837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b409a96-8219-4fe5-9207-b459e891a3f8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:43.345837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b409a96-8219-4fe5-9207-b459e891a3f8 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:43.345837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 437/873] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:43 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:43.346895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:43.349207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0044f9-4d6d-429b-a9e0-99770503a7d1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:43.349409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0044f9-4d6d-429b-a9e0-99770503a7d1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:43.349686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0044f9-4d6d-429b-a9e0-99770503a7d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:43.374268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-ac0044f9-4d6d-429b-a9e0-99770503a7d1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:43.375468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ac0044f9-4d6d-429b-a9e0-99770503a7d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:43.460038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 438/874] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:43 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:43.463877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09d2ec0-e5c3-4cf5-9519-384e17f39118 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:43.499471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:43.500472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:43.500937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:43.501381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:43.502251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:43.503832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:43.521016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 437/875] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:41:43 2022] POST /identity/v3/services => generated 251 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:43.733210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0044f9-4d6d-429b-a9e0-99770503a7d1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:43.833836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:43.850957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0044f9-4d6d-429b-a9e0-99770503a7d1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:43.893472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:43.894103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:43.894379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:43.894644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:43.895275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:43.896671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:43.907045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:43.908178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:43.909282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 439/876] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:43 2022] GET /identity/v3/services/object-store => generated 93 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:43.916544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:43.961494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:44.026841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc24d008-07af-4799-a95f-c7be6907c4a4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:44.066611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bea68df-c820-4ce0-b7cc-cbe5b5981e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:44.067583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bea68df-c820-4ce0-b7cc-cbe5b5981e17 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:44.068116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bea68df-c820-4ce0-b7cc-cbe5b5981e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:44.068620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bea68df-c820-4ce0-b7cc-cbe5b5981e17 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:44.069449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bea68df-c820-4ce0-b7cc-cbe5b5981e17 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:44.070013 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bea68df-c820-4ce0-b7cc-cbe5b5981e17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:44.071568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bea68df-c820-4ce0-b7cc-cbe5b5981e17 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:44.081422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 438/877] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:43 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:44.083865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:44.196968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e3c02d-cd42-4050-8a6a-d991597f1d70 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:44.231876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500a66fb-1130-4c95-8c51-ce04dadd51d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:44.232652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a66fb-1130-4c95-8c51-ce04dadd51d5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:44.232928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a66fb-1130-4c95-8c51-ce04dadd51d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:44.233192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500a66fb-1130-4c95-8c51-ce04dadd51d5 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:44.233810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500a66fb-1130-4c95-8c51-ce04dadd51d5 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:44.234104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500a66fb-1130-4c95-8c51-ce04dadd51d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:44.235459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500a66fb-1130-4c95-8c51-ce04dadd51d5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:44.240319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 440/878] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:44 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:44.277722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c7417-eb70-466b-8cfd-01a19cbe7d87 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:44.277957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c7417-eb70-466b-8cfd-01a19cbe7d87 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:44.278205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c7417-eb70-466b-8cfd-01a19cbe7d87 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:44.279896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 439/879] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:44.285677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12931ae-c991-43a2-aaca-d176358f7668 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:44.285963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12931ae-c991-43a2-aaca-d176358f7668 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:44.286237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12931ae-c991-43a2-aaca-d176358f7668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:44.306463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d12931ae-c991-43a2-aaca-d176358f7668 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:44.307691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d12931ae-c991-43a2-aaca-d176358f7668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:44.310717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:44.384015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 441/880] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:44 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:44.654775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301c7417-eb70-466b-8cfd-01a19cbe7d87 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:44.759766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301c7417-eb70-466b-8cfd-01a19cbe7d87 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:44.761526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:44.799412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6d2fc6-8eb8-40be-b8ae-3002a9c1f029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:44.800491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6d2fc6-8eb8-40be-b8ae-3002a9c1f029 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:44.800971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6d2fc6-8eb8-40be-b8ae-3002a9c1f029 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:44.801490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6d2fc6-8eb8-40be-b8ae-3002a9c1f029 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:44.802352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6d2fc6-8eb8-40be-b8ae-3002a9c1f029 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:44.803899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6d2fc6-8eb8-40be-b8ae-3002a9c1f029 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:44.826428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 440/881] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:41:44 2022] POST /identity/v3/services => generated 269 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:41:45.393500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96055f15-087d-4213-9f85-c1394a859322 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:45.394016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96055f15-087d-4213-9f85-c1394a859322 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:45.394119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96055f15-087d-4213-9f85-c1394a859322 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:45.395627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 442/882] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:45.400926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea110d-a92d-4961-9588-45acaf761dbd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:45.401215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea110d-a92d-4961-9588-45acaf761dbd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:45.401491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea110d-a92d-4961-9588-45acaf761dbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:45.421100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-56ea110d-a92d-4961-9588-45acaf761dbd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:45.422298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-56ea110d-a92d-4961-9588-45acaf761dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:45.497517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 441/883] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:45 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:45.771687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96055f15-087d-4213-9f85-c1394a859322 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:45.890027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96055f15-087d-4213-9f85-c1394a859322 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:45.907894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eae77b5-2752-4c42-8aa1-4819b4260ef6 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:45.908450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eae77b5-2752-4c42-8aa1-4819b4260ef6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:45.908536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eae77b5-2752-4c42-8aa1-4819b4260ef6 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:45.910093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 442/884] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:45.914968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e700ef-f648-4bd8-b910-2420fb8a4414 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:45.915252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e700ef-f648-4bd8-b910-2420fb8a4414 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:45.915525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e700ef-f648-4bd8-b910-2420fb8a4414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:45.928426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:45.929436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:45.929978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:45.930449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:45.935179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-38e700ef-f648-4bd8-b910-2420fb8a4414 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:45.936809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-38e700ef-f648-4bd8-b910-2420fb8a4414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:45.941987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:45.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:45.948164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:41:45.949028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:45.952636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 443/885] 10.222.0.42 () {56 vars in 1120 bytes} [Fri Sep 16 01:41:45 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:45.952792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:45.958438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.030321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 443/886] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:45 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:46.072377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f9fa0c-ba89-4a2d-b685-f5d5f40a407d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.109060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.110049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.110521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.110992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.111841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:46.112376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.113925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.124124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 444/887] 10.222.0.42 () {56 vars in 1115 bytes} [Fri Sep 16 01:41:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:46.133059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e700ef-f648-4bd8-b910-2420fb8a4414 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.156288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:46.238126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e700ef-f648-4bd8-b910-2420fb8a4414 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.273318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.273944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.274221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.274487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.285443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.286820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:46.301878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:46.302938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:46.304514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:46.304514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:46.304514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:46.304514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 444/888] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:46.309525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.326986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.382780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:46.411982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:46.436296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b15917-baf9-4275-b1cc-07f4ef5953a2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.448565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e03823-d107-4374-845d-f45bafb6c207 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.472380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d050e75f-935b-4017-9603-cc2556eb0e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.472998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050e75f-935b-4017-9603-cc2556eb0e9b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.473268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050e75f-935b-4017-9603-cc2556eb0e9b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.473521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d050e75f-935b-4017-9603-cc2556eb0e9b None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.474159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d050e75f-935b-4017-9603-cc2556eb0e9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:46.474453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d050e75f-935b-4017-9603-cc2556eb0e9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.476054 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d050e75f-935b-4017-9603-cc2556eb0e9b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.488021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e22658-5faa-410e-adac-3770eccefc3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.488890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e22658-5faa-410e-adac-3770eccefc3a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.489330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e22658-5faa-410e-adac-3770eccefc3a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.489629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 445/889] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:46.489993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e22658-5faa-410e-adac-3770eccefc3a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.490578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e22658-5faa-410e-adac-3770eccefc3a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.490984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33456c3-589a-4316-80e1-f162edbc2b13 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.491270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33456c3-589a-4316-80e1-f162edbc2b13 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.491536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33456c3-589a-4316-80e1-f162edbc2b13 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.491928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e22658-5faa-410e-adac-3770eccefc3a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.492843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 446/890] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:46.497252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33456c3-589a-4316-80e1-f162edbc2b13 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-72e22658-5faa-410e-adac-3770eccefc3a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:46.501798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:46.502804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:46.503676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:46.507847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 445/891] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:46.509198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4870a173-9f1b-4cf6-8941-2c5e0a8cdfb4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.509478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4870a173-9f1b-4cf6-8941-2c5e0a8cdfb4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.509748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4870a173-9f1b-4cf6-8941-2c5e0a8cdfb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.530350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4870a173-9f1b-4cf6-8941-2c5e0a8cdfb4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:46.531515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4870a173-9f1b-4cf6-8941-2c5e0a8cdfb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:46.603851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 446/892] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:46 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:46.604856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4870a173-9f1b-4cf6-8941-2c5e0a8cdfb4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.613179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33456c3-589a-4316-80e1-f162edbc2b13 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.635051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:46.652393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.654239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.654743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.655214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.662880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.664544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.671484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:46.678717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:46.679690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:46.680665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 447/893] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:46.687063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.718858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4870a173-9f1b-4cf6-8941-2c5e0a8cdfb4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.759306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.760354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.760866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.761597 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.762512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:46.763157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.764792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.770404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 447/894] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:46.781043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.811311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cc4831-8ce6-4154-8938-d12ebe19584f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.850769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.851162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.851502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.851835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.852637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:46.853003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.854747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.876840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 448/895] 10.222.0.42 () {56 vars in 1149 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:46.881905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.917420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4698731f-d22c-421c-92c2-72d5a51f0f99 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:46.925540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:46.947173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:46.957464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:46.958212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:46.959296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:46.959579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:46.960235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:46.960532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:46.961882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:46.969115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 448/896] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:46.969956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:46.991049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8558d54-8a19-4917-9c28-d98f3c7eea7f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.031607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6434d418-47dc-4fb6-9dbc-7b8e52c032d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.032352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6434d418-47dc-4fb6-9dbc-7b8e52c032d5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.032626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6434d418-47dc-4fb6-9dbc-7b8e52c032d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.032891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6434d418-47dc-4fb6-9dbc-7b8e52c032d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.082169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201ea781-1009-4634-b9f8-f59f1b5ee756 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.106324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6434d418-47dc-4fb6-9dbc-7b8e52c032d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.107873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6434d418-47dc-4fb6-9dbc-7b8e52c032d5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.116910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.117511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.117811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.118059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.118696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.120165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.135057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 449/897] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:41:46 2022] POST /identity/v3/services => generated 241 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:47.136354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.150370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:47.210512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:47.220341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 449/898] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:46 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:41:47.221931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb4995-eac3-4c43-852a-8df0d925a2c8 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.222220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb4995-eac3-4c43-852a-8df0d925a2c8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.222496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bb4995-eac3-4c43-852a-8df0d925a2c8 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.223779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 450/899] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:47.226122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bb4995-eac3-4c43-852a-8df0d925a2c8 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.279392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786380c1-80b2-4f56-9ba4-b8537de94793 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.318837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27b3d33-285a-4bfb-9502-7ba06d60fb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.319446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b3d33-285a-4bfb-9502-7ba06d60fb09 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.319723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b3d33-285a-4bfb-9502-7ba06d60fb09 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.320069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b3d33-285a-4bfb-9502-7ba06d60fb09 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.320767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27b3d33-285a-4bfb-9502-7ba06d60fb09 None admin] RBAC: Adding query filter params (interface=public, service_id=6a93b272322d49a889c162cb6ce9523c, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:47.321617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27b3d33-285a-4bfb-9502-7ba06d60fb09 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.322458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27b3d33-285a-4bfb-9502-7ba06d60fb09 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.329175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 450/900] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/endpoints?service_id=6a93b272322d49a889c162cb6ce9523c&interface=public®ion_id=RegionOne => generated 194 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:47.330375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f8bd88-3ea9-47cd-a358-7c03d0b8d955 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.330652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f8bd88-3ea9-47cd-a358-7c03d0b8d955 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.330921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f8bd88-3ea9-47cd-a358-7c03d0b8d955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.351515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-39f8bd88-3ea9-47cd-a358-7c03d0b8d955 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:47.352881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-39f8bd88-3ea9-47cd-a358-7c03d0b8d955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:47.357391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bb4995-eac3-4c43-852a-8df0d925a2c8 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.394224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.394854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.395140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.395412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.402496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:47.407202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.408944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:47.426603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:47.427598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:47.428531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:47.428531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:47.428531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:47.432091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 451/901] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:47.433256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.435194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 451/902] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:47.436216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f8bd88-3ea9-47cd-a358-7c03d0b8d955 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.490453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:47.557014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f49058-c5ef-49b3-8fe2-fe35ee5b7a7c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.569104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f8bd88-3ea9-47cd-a358-7c03d0b8d955 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.594316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.595122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.595593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.595976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.596688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.598154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.606945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 452/903] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/services => generated 1497 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:41:47.613374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.614225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.614650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.615063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.615876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:47.616359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.617936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.632599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 452/904] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:47.642380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.705538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:47.714782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.759129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75abee4e-d223-49ad-b673-841fcaa8ad82 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.798023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.798692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.798969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.799240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.813027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.814748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:47.829463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:47.830408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:47.831301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 453/905] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:47.836804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.853384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0510c49d-4635-4a72-a0bd-53f30c70990c None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.855296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:47.892490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:47.893190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:47.893601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:47.893937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:47.894590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:47.896134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:47.906083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:47.908177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:47.909370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 453/906] 10.222.0.42 () {56 vars in 1132 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:47.913634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:47.974232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b9b98e-8fc8-4fc6-9328-a1ed6c718a8e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:47.991639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:48.020077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:48.020840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:48.021186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:48.021545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:48.022569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:48.022968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:48.025201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:48.029799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a224aab5-de87-409a-822f-8d668d2fd9d6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:48.045736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 454/907] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:48.046846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:48.076333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:48.077105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:48.077346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:48.077610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:48.078264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:48.078621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:48.080054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:48.090600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 454/908] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:47 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:48.091616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:48.124244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:48.179806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a98aa1-cc06-4c30-9bca-cf9962f8086e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:48.210090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc908c09-7243-4e07-8d6a-a26b9ba24cb8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:48.215413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:48.216041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:48.216317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:48.216580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:48.217299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] RBAC: Adding query filter params (role.id=18761cd58fe9422485f4b8e316845930, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=f58ec160611e446193b43582a8155284) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:48.217600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:48.219013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:48.227971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 455/909] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:48 2022] GET /identity/v3/role_assignments?user.id=f58ec160611e446193b43582a8155284&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=18761cd58fe9422485f4b8e316845930 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:48.254661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:48.255570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:48.256077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:48.256488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:48.257381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:48.257830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:48.259511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:48.266142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 455/910] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:48 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:48.271994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:48.319188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:48.389670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7f318e-f695-46ac-9259-cdf16cc87821 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:48.431182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c35602-2563-42b5-9f77-2872060f2bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:48.432091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c35602-2563-42b5-9f77-2872060f2bdc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:48.432516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c35602-2563-42b5-9f77-2872060f2bdc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:48.432947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c35602-2563-42b5-9f77-2872060f2bdc None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:48.434764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c35602-2563-42b5-9f77-2872060f2bdc None admin] RBAC: Adding query filter params (interface=public, service_id=247d16260e304e20a93b91e196a6153a, region_id=RegionOne) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:48.435245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c35602-2563-42b5-9f77-2872060f2bdc None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:48.436962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c35602-2563-42b5-9f77-2872060f2bdc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:48.446424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 456/911] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:41:48 2022] GET /identity/v3/endpoints?service_id=247d16260e304e20a93b91e196a6153a&interface=public®ion_id=RegionOne => generated 194 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:48.452753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:48.557478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:48.566024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae4a48b-30f2-4894-ab96-d52ee06409f4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:48.601243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2642a9-c538-4ba7-9217-21488f3a8254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:48.601839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2642a9-c538-4ba7-9217-21488f3a8254 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:48.602111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2642a9-c538-4ba7-9217-21488f3a8254 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:48.602375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2642a9-c538-4ba7-9217-21488f3a8254 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:48.602968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2642a9-c538-4ba7-9217-21488f3a8254 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:48.604347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2642a9-c538-4ba7-9217-21488f3a8254 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:48.611061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 456/912] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:41:48 2022] GET /identity/v3/services => generated 1497 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:41:49.406641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9202dfd3-752f-46ed-99bc-87af8b4ffeeb None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:49.407639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9202dfd3-752f-46ed-99bc-87af8b4ffeeb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:49.408175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9202dfd3-752f-46ed-99bc-87af8b4ffeeb None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:49.409770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 457/913] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:49.415471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dde85-8378-4f37-824c-c14170582830 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:49.415900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dde85-8378-4f37-824c-c14170582830 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:49.416029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dde85-8378-4f37-824c-c14170582830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:49.436154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e49dde85-8378-4f37-824c-c14170582830 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:49.437334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e49dde85-8378-4f37-824c-c14170582830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:49.507849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 457/914] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:49.786546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9202dfd3-752f-46ed-99bc-87af8b4ffeeb None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:49.903332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9202dfd3-752f-46ed-99bc-87af8b4ffeeb None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:49.906521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:49.952427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4354ec8-0282-49ab-9d56-9ff9e23f9427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:49.953072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4354ec8-0282-49ab-9d56-9ff9e23f9427 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:49.953350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4354ec8-0282-49ab-9d56-9ff9e23f9427 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:49.953620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4354ec8-0282-49ab-9d56-9ff9e23f9427 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:49.954211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4354ec8-0282-49ab-9d56-9ff9e23f9427 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:49.955576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4354ec8-0282-49ab-9d56-9ff9e23f9427 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-c4354ec8-0282-49ab-9d56-9ff9e23f9427 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:49.964982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:49.966529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:49.967928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 458/915] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:49 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:49.972646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49dde85-8378-4f37-824c-c14170582830 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:50.051100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd25b2f-85d4-44db-a79c-a6071d56c5d9 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.051415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd25b2f-85d4-44db-a79c-a6071d56c5d9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.051576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd25b2f-85d4-44db-a79c-a6071d56c5d9 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.053163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 459/916] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:50.058632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab84673-482d-4d27-803e-4ee0b50e0183 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.059405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab84673-482d-4d27-803e-4ee0b50e0183 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.059745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab84673-482d-4d27-803e-4ee0b50e0183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.068309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:50.082055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49dde85-8378-4f37-824c-c14170582830 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:50.084266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0ab84673-482d-4d27-803e-4ee0b50e0183 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:50.086427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0ab84673-482d-4d27-803e-4ee0b50e0183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:50.118776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:50.119806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.120351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.120798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.122194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:50.122693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:50.124479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:50.129933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 458/917] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:49 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:50.134967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:50.173305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 460/918] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:50.241963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf937cb-e30b-4792-9814-e4ba7a633686 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:50.278149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:50.278748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.279029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.279299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.279921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:50.280245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:50.281615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:50.286940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 459/919] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:50 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:50.293105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab84673-482d-4d27-803e-4ee0b50e0183 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:50.398766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:50.405125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab84673-482d-4d27-803e-4ee0b50e0183 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:50.441480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:50.442100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.442349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.442683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.443389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] RBAC: Adding query filter params (interface=public, service_id=72b004820f294b34be2258cee35b856d, region_id=RegionOne) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:50.443693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:50.445088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:50.450657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 461/920] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:41:50 2022] GET /identity/v3/endpoints?service_id=72b004820f294b34be2258cee35b856d&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:50.457224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:50.469167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:50.478286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:50.583534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee45ec4-6095-4af7-9206-3fe0bcc75415 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:50.591398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31842997-c295-4778-a17e-cc19e5b7364e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:50.620467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1cf8b7-9fa0-4895-a402-9eccd5fc789b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:50.621156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1cf8b7-9fa0-4895-a402-9eccd5fc789b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.622114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1cf8b7-9fa0-4895-a402-9eccd5fc789b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.622594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1cf8b7-9fa0-4895-a402-9eccd5fc789b None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.623144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1cf8b7-9fa0-4895-a402-9eccd5fc789b None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:50.624542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1cf8b7-9fa0-4895-a402-9eccd5fc789b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:50.627971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:50.628583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.628848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.629118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.629718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:50.631071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-fa1cf8b7-9fa0-4895-a402-9eccd5fc789b None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:50.633767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:50.634690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:50.635523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 462/921] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:50 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:50.638171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63c58bc-2258-4049-a7f5-6835d06f5fbe None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.638171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63c58bc-2258-4049-a7f5-6835d06f5fbe None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.638171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63c58bc-2258-4049-a7f5-6835d06f5fbe None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.639432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 463/922] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:50.652386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 460/923] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:41:50 2022] GET /identity/v3/services => generated 1497 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:41:50.652723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30abae8b-54be-4787-bb3c-b7672e0283a3 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.652723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30abae8b-54be-4787-bb3c-b7672e0283a3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.652723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30abae8b-54be-4787-bb3c-b7672e0283a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.653440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:50.670349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-30abae8b-54be-4787-bb3c-b7672e0283a3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:50.671733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-30abae8b-54be-4787-bb3c-b7672e0283a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:50.693607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:50.751911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:50.764617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 464/924] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:50.775161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ab3e95-a6f7-4ca0-ab70-74161c18dd05 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:50.815072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:50.815695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.815995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.816326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.816975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:50.817325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:50.818731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:50.824800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 461/925] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:50 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:50.833345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30abae8b-54be-4787-bb3c-b7672e0283a3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:50.950515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30abae8b-54be-4787-bb3c-b7672e0283a3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:50.987011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cf028b-1663-4118-beeb-bfb966574d16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:50.987624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf028b-1663-4118-beeb-bfb966574d16 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:50.987902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf028b-1663-4118-beeb-bfb966574d16 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:50.988195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cf028b-1663-4118-beeb-bfb966574d16 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:50.988886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cf028b-1663-4118-beeb-bfb966574d16 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:50.989121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cf028b-1663-4118-beeb-bfb966574d16 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:50.990481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cf028b-1663-4118-beeb-bfb966574d16 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:50.995894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 465/926] 10.222.0.42 () {56 vars in 1114 bytes} [Fri Sep 16 01:41:50 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:51.002105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.014774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cecb9b-58a0-4e12-b200-8ca57edb6dce None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.014774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cecb9b-58a0-4e12-b200-8ca57edb6dce None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.014774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cecb9b-58a0-4e12-b200-8ca57edb6dce None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.014774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 466/927] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:51.019513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf3fd2-1afc-4e0f-9ec0-486fbba22af4 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.019811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf3fd2-1afc-4e0f-9ec0-486fbba22af4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.020113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf3fd2-1afc-4e0f-9ec0-486fbba22af4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.038478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7fbf3fd2-1afc-4e0f-9ec0-486fbba22af4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:51.039677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7fbf3fd2-1afc-4e0f-9ec0-486fbba22af4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:51.054581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:51.112844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 467/928] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:51.113922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbf3fd2-1afc-4e0f-9ec0-486fbba22af4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.133297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a35ecc-98f3-4e58-badb-4026a5a890a5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.182057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.183243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.184129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.184583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.185274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:51.186644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:51.206295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:51.212620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 462/929] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:41:51 2022] POST /identity/v3/endpoints => generated 346 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:51.247275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbf3fd2-1afc-4e0f-9ec0-486fbba22af4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.296967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04494146-2125-4aea-b466-70944bb7693a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.298077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04494146-2125-4aea-b466-70944bb7693a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.298642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04494146-2125-4aea-b466-70944bb7693a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.299144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04494146-2125-4aea-b466-70944bb7693a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.313118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04494146-2125-4aea-b466-70944bb7693a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-04494146-2125-4aea-b466-70944bb7693a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:51.318768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:51.319707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:41:51.320602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:51.323466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 468/930] 10.222.0.42 () {56 vars in 1120 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:51.329713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.396885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04494146-2125-4aea-b466-70944bb7693a None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.403977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:51.453634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a05d978-15d5-4b06-8f89-1557ba302fb2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.492866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.493477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.493664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.493933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.494610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:51.495202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:51.497193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:51.511983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 463/931] 10.222.0.42 () {56 vars in 1115 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:51.514357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.522963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04494146-2125-4aea-b466-70944bb7693a None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.561444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.562083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.562363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.562679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.563340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:51.564672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:51.574810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:51.576011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:51.576909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 469/932] 10.222.0.42 () {56 vars in 1132 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:51.581848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.584898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:51.629176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07e140f-2ac0-499a-a0ba-c9a32c72a6eb None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.669492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.670367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.670873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.671291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.673090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:51.685336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:51.687143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:51.703158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:51.704153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:51.705033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:51.705033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:51.705033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:51.711539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 464/933] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:51.712634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.713983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8319f39-d026-413e-9902-4e299cae1e5b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.759110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.759641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.759921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.760225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.761551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:51.762388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:51.764289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:51.770626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 470/934] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:51.775706 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.833677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3304dbf8-9f20-4fb4-a231-15583a119c15 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.856106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:51.876590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.876834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.877181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.877513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.878219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:51.878585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:51.880147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:51.898059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 465/935] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:51.899097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:51.918826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f573af8-3ccd-4f87-b584-3f5165dab8f9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:51.950488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:51.957688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:51.958245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:51.958597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:51.958962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:51.959801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:51.960056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:51.961542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:51.969905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 471/936] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:51.975509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:52.030965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75537243-6119-4b1e-8bbe-c27f6aa2e109 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:52.070302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:52.071068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.071467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.071744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.082567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:52.083637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa6b845-5cf7-4264-ae80-7ee2844feb6e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:52.084182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:52.099228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:52.100403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:52.101367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:52.104938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 466/937] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:41:51 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:52.110542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:52.124692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:52.125395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.125766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.126109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.126853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:52.128315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:52.148443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:52.160150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 472/938] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:41:51 2022] POST /identity/v3/endpoints => generated 347 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:41:52.227453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49db4a64-18fc-422c-a0e4-81adfbec1336 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:52.275675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:52.276316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.276587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.276849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.277657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:52.277957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:52.279391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:52.304325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 467/939] 10.222.0.42 () {56 vars in 1149 bytes} [Fri Sep 16 01:41:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:52.306036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:52.370346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:52.430775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9c30e0-ab4f-4128-a5b2-947f9201ec35 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:52.467498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:52.468195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.468475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.468747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.480020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:52.481505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:52.496153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:52.497189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:52.498235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:52.498235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:52.498235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:52.498235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 473/940] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:52.503047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:52.597848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:52.613456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3179ed62-5413-42ba-9f04-612a41234dd8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:52.649846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:52.650449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.650722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.650987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.651614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:52.651905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:52.653969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:52.666091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 468/941] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:52.672386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:52.776186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61778ebb-5916-4ed9-ac37-92818320dad5 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:52.784334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:52.813133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6db3ce-81dc-4d1e-9063-da21babd680d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:52.813760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6db3ce-81dc-4d1e-9063-da21babd680d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.814033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6db3ce-81dc-4d1e-9063-da21babd680d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.814298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6db3ce-81dc-4d1e-9063-da21babd680d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.821372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6db3ce-81dc-4d1e-9063-da21babd680d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:52.822799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6db3ce-81dc-4d1e-9063-da21babd680d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-cd6db3ce-81dc-4d1e-9063-da21babd680d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:52.834079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:52.834962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:52.836250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 474/942] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:52.842131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:52.961158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58adf516-b48e-4796-9090-3acdb301ccae None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:52.971684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12abfdf-fcb7-4ab0-abf3-ff04ed32f3f5 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.971987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12abfdf-fcb7-4ab0-abf3-ff04ed32f3f5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.972250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12abfdf-fcb7-4ab0-abf3-ff04ed32f3f5 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.973840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 475/943] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:52.978796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b839faa0-c291-498e-80d2-dada5614bde6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:52.979076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b839faa0-c291-498e-80d2-dada5614bde6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:52.979354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b839faa0-c291-498e-80d2-dada5614bde6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:52.999304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b839faa0-c291-498e-80d2-dada5614bde6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:53.000521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b839faa0-c291-498e-80d2-dada5614bde6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:53.004751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:53.005638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:53.006093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:53.006543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:53.007403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:53.008313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:53.010272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:53.033163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 469/944] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:53.035434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:53.049965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:53.092172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 476/945] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:52 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:53.151276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24de41e0-0177-40c5-a3a8-e92ca2f69f25 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:53.188261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:53.188932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:53.189256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:53.189523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/f58ec160611e446193b43582a8155284/roles/18761cd58fe9422485f4b8e316845930` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:53.191887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=f58ec160611e446193b43582a8155284, role_id=18761cd58fe9422485f4b8e316845930)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:53.193397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:53.212912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 470/946] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:41:53 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/f58ec160611e446193b43582a8155284/roles/18761cd58fe9422485f4b8e316845930 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:41:53.384199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b839faa0-c291-498e-80d2-dada5614bde6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:53.491398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:53.500026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b839faa0-c291-498e-80d2-dada5614bde6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:53.536675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c8d3f5-dfda-4fd3-985b-1131b25a8fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:53.537296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c8d3f5-dfda-4fd3-985b-1131b25a8fba None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:53.537568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c8d3f5-dfda-4fd3-985b-1131b25a8fba None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:53.537833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c8d3f5-dfda-4fd3-985b-1131b25a8fba None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:53.538407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c8d3f5-dfda-4fd3-985b-1131b25a8fba None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:53.539752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c8d3f5-dfda-4fd3-985b-1131b25a8fba None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-80c8d3f5-dfda-4fd3-985b-1131b25a8fba None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:53.549181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:53.550413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:53.551420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:53.553674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 477/947] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:53 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:53.560452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:53.593353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af498f09-d0b1-430c-8503-6a05beda95a3 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:53.593667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af498f09-d0b1-430c-8503-6a05beda95a3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:53.593902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af498f09-d0b1-430c-8503-6a05beda95a3 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:53.595571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 478/948] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:53.601178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd36b57-17e8-40f0-be5a-d3c4ab562bcb None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:53.601464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd36b57-17e8-40f0-be5a-d3c4ab562bcb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:53.601742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd36b57-17e8-40f0-be5a-d3c4ab562bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:53.621717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-bdd36b57-17e8-40f0-be5a-d3c4ab562bcb None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:53.623053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bdd36b57-17e8-40f0-be5a-d3c4ab562bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:53.634878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:53.681084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43efcaf-1fc0-4836-b653-100667e43bfd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:53.702268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 479/949] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:53 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:53.721618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:53.722208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:53.722479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:53.722743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:53.723362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:53.723652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:53.725030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:53.730017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 471/950] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:53 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:53.736106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd36b57-17e8-40f0-be5a-d3c4ab562bcb None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:53.832461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:53.846541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd36b57-17e8-40f0-be5a-d3c4ab562bcb None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:53.884234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:53.885242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:53.885731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:53.886447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:53.887400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:53.887917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:53.889494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:53.894826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 480/951] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:53 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:53.901269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:53.996776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:54.009081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5370c656-3da0-4e09-b451-cd88f8f8e260 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:54.049039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:54.055324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:54.055928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:54.056228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:54.056493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:54.057156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:54.058520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:54.087277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 472/952] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:41:53 2022] POST /identity/v3/endpoints => generated 349 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:54.123796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f3a18a-6e60-48e4-ac35-7ff1ca2edb5c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:54.174122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:54.175322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:54.175810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:54.176358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:54.177230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:54.178917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:54.182010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING keystone.server.flask.common [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 16 01:41:54.204709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.sql.core [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 16 01:41:54.204709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 16 01:41:54.204709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [parameters: {'id': '3d46416bed8e4e1d8a00b49e7abdc5af', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 16 01:41:54.204709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81424) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 3d46416bed8e4e1d8a00b49e7abdc5af.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 3d46416bed8e4e1d8a00b49e7abdc5af. Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application conn.query(q) Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 01:41:54.213986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result.read() Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 16 01:41:54.214936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application next(self.gen) Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application yield resource Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application next(self.gen) Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.session.rollback() Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.force_reraise() Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise self.value Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application session.commit() Sep 16 01:41:54.215986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._prepare_impl() Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.session.flush() Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._flush(objects) Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application compat.raise_( Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application flush_context.execute() Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 16 01:41:54.216907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rec.execute(self) Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 16 01:41:54.218694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application conn.query(q) Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result.read() Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 01:41:54.220279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [parameters: {'id': '3d46416bed8e4e1d8a00b49e7abdc5af', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:54.222567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:54.224509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 3d46416bed8e4e1d8a00b49e7abdc5af. Sep 16 01:41:54.226801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:54.230217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 481/953] 10.222.0.42 () {60 vars in 1140 bytes} [Fri Sep 16 01:41:53 2022] POST /identity/v3/projects => generated 262 bytes in 234 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:41:54.237182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:54.353993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c47492b-6043-44cb-a470-bc150e99e577 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:54.393476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4141eca3-798c-4913-8128-eaacd0e6d214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:54.394584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141eca3-798c-4913-8128-eaacd0e6d214 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:54.395105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141eca3-798c-4913-8128-eaacd0e6d214 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:54.395584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141eca3-798c-4913-8128-eaacd0e6d214 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:54.404093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4141eca3-798c-4913-8128-eaacd0e6d214 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:54.405985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4141eca3-798c-4913-8128-eaacd0e6d214 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-4141eca3-798c-4913-8128-eaacd0e6d214 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:54.419019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:54.420055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:54.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:54.423456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 473/954] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:54.429700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:54.434838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:54.505243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52db2c3-a95f-4999-88e9-4d9baa3c0da3 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:54.506162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52db2c3-a95f-4999-88e9-4d9baa3c0da3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:54.506641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52db2c3-a95f-4999-88e9-4d9baa3c0da3 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:54.509465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 474/955] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:54 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:54.520327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05072b-be08-4321-8fdd-bb4815e55209 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:54.521247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05072b-be08-4321-8fdd-bb4815e55209 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:54.522049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05072b-be08-4321-8fdd-bb4815e55209 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:54.544108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541f96be-2543-4b6e-b1f3-9fb8028deb2e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:54.548258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0e05072b-be08-4321-8fdd-bb4815e55209 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:54.549696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0e05072b-be08-4321-8fdd-bb4815e55209 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:54.582464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:54.583737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:54.584392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:54.584658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:54.585331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:54.585641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:54.587386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:54.605146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:54.619837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 482/956] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:54 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:54.637382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 475/957] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:54 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:54.919437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:54.989689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:55.034315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feea4cc5-d8b3-42b3-a061-587c323e7732 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:55.072387 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:55.073053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:55.073336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:55.073612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] PATH_INFO: `/v3/services/compute` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:55.074216 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:55.076033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:55.085813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:55.086886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:55.087921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 483/958] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:54 2022] GET /identity/v3/services/compute => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:55.095772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05072b-be08-4321-8fdd-bb4815e55209 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:55.200379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05072b-be08-4321-8fdd-bb4815e55209 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:55.236640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b125a2f4-c52e-4edc-91eb-65afa78a259f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:55.237590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125a2f4-c52e-4edc-91eb-65afa78a259f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:55.238100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125a2f4-c52e-4edc-91eb-65afa78a259f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:55.238574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b125a2f4-c52e-4edc-91eb-65afa78a259f None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:55.239411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b125a2f4-c52e-4edc-91eb-65afa78a259f None admin] RBAC: Adding query filter params (name=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:55.239911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b125a2f4-c52e-4edc-91eb-65afa78a259f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:55.241492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b125a2f4-c52e-4edc-91eb-65afa78a259f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:55.252054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 476/959] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:55 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:55.253837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:55.360166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93261cb1-aade-4c9a-90f3-b03a727e4e8f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:55.393065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:55.401913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79aa92bd-f861-412f-8296-fb758f4ce93a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:55.402726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa92bd-f861-412f-8296-fb758f4ce93a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:55.403617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa92bd-f861-412f-8296-fb758f4ce93a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:55.404183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79aa92bd-f861-412f-8296-fb758f4ce93a None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:55.405039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa92bd-f861-412f-8296-fb758f4ce93a None admin] RBAC: Adding query filter params (type=compute) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:55.405346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa92bd-f861-412f-8296-fb758f4ce93a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:55.406750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79aa92bd-f861-412f-8296-fb758f4ce93a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:55.412751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 484/960] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:41:55 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:55.516084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9fea2-ee5c-41a9-8687-c2e3c06c2cd5 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:55.516276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9fea2-ee5c-41a9-8687-c2e3c06c2cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:55.516539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b9fea2-ee5c-41a9-8687-c2e3c06c2cd5 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:55.518139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 477/961] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:55.523805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b50a0-39f4-4702-a7a1-480c9e5bbef6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:55.524118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b50a0-39f4-4702-a7a1-480c9e5bbef6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:55.524390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b50a0-39f4-4702-a7a1-480c9e5bbef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:55.545103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-785b50a0-39f4-4702-a7a1-480c9e5bbef6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:55.546659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-785b50a0-39f4-4702-a7a1-480c9e5bbef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:55.623345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 485/962] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:55 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:55.899096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b9fea2-ee5c-41a9-8687-c2e3c06c2cd5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:55.945041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:56.006410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b9fea2-ee5c-41a9-8687-c2e3c06c2cd5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:56.053239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:56.054252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.054736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.055222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.066130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:41:56.071285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:56.072234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:41:56.073076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:56.076137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 478/963] 10.222.0.42 () {56 vars in 1120 bytes} [Fri Sep 16 01:41:55 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:56.084006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b50a0-39f4-4702-a7a1-480c9e5bbef6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:56.191413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b50a0-39f4-4702-a7a1-480c9e5bbef6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:56.216707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:56.230225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe67fd5-9e84-4394-b79d-1cce6e27228c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:56.231890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe67fd5-9e84-4394-b79d-1cce6e27228c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.232373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe67fd5-9e84-4394-b79d-1cce6e27228c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.232854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe67fd5-9e84-4394-b79d-1cce6e27228c None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.233724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe67fd5-9e84-4394-b79d-1cce6e27228c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:56.234217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe67fd5-9e84-4394-b79d-1cce6e27228c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:56.235736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe67fd5-9e84-4394-b79d-1cce6e27228c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:56.244814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 486/964] 10.222.0.42 () {56 vars in 1115 bytes} [Fri Sep 16 01:41:56 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:56.251150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:56.366867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad6045-81d6-458f-a93b-0ae25d44d567 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:56.368731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:56.395248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0810e42b-f958-48d9-86b7-de41e8b292ad None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.395537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0810e42b-f958-48d9-86b7-de41e8b292ad None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.395811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0810e42b-f958-48d9-86b7-de41e8b292ad None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.397410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 487/965] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:41:56.402200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f0983-fabd-4662-9053-3c9731317d3d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.402534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f0983-fabd-4662-9053-3c9731317d3d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.402809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f0983-fabd-4662-9053-3c9731317d3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.412441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:56.413375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.413836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.414302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.421890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-897f0983-fabd-4662-9053-3c9731317d3d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:56.423079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-897f0983-fabd-4662-9053-3c9731317d3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:56.427886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:56.429633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:56.452098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:56.453061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:56.453948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:56.453948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:56.453948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:56.456371 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 479/966] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:56.461954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:56.503097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 488/967] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:56 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:41:56.580352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a9914-b953-4480-a3c8-e91e02a07ace None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:56.615709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:56.616567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.617076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.617493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.618271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:56.618711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:56.620228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:56.633527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 480/968] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:56.640115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f0983-fabd-4662-9053-3c9731317d3d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:56.745624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f0983-fabd-4662-9053-3c9731317d3d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:56.761533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:56.783458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:56.784124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.784396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.784658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.792725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:56.792725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:56.806176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:56.807145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:56.808126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 489/969] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:41:56 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:56.814175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:56.820631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:56.839199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:56.933267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b268a5-6292-45b8-9ef8-e273f676192b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:56.942127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f74750-a7e9-4c93-a931-eb271606ea39 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:56.976365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:56.976365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.976365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.976729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.977154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:56.978522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:56.980273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:56.981664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:56.981979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:56.982246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:56.982964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:56.983258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:56.984676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:56.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:56.990300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:41:56.991137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:56.994067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 481/970] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:41:56 2022] GET /identity/v3/services/object-store => generated 93 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:57.000067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.009846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 490/971] 10.222.0.42 () {56 vars in 1149 bytes} [Fri Sep 16 01:41:56 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:57.017009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.054319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.094894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.115552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e7ed17-6225-41c7-ae5e-ee875a34db7b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:57.140343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f65cc3-4624-46cf-aa13-74752d1d2072 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:57.151459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1a74cc-49f8-4ba2-af39-a135f14ae8d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:57.152096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a74cc-49f8-4ba2-af39-a135f14ae8d2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.152454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a74cc-49f8-4ba2-af39-a135f14ae8d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.153213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1a74cc-49f8-4ba2-af39-a135f14ae8d2 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.153846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1a74cc-49f8-4ba2-af39-a135f14ae8d2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:57.154146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1a74cc-49f8-4ba2-af39-a135f14ae8d2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:57.155510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1a74cc-49f8-4ba2-af39-a135f14ae8d2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:57.161100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 482/972] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:56 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:57.161100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c574125f-8605-4906-b0ed-0ecb3fe61f80 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.161100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c574125f-8605-4906-b0ed-0ecb3fe61f80 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.161100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c574125f-8605-4906-b0ed-0ecb3fe61f80 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.161100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 483/973] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:57 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:57.166542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c574125f-8605-4906-b0ed-0ecb3fe61f80 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.182831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:57.183781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.183781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.184251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.255682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:57.258150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:57.288621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c574125f-8605-4906-b0ed-0ecb3fe61f80 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:57.291732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.330521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56626ebb-462f-499f-a7ac-a787f29c4472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:57.330746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56626ebb-462f-499f-a7ac-a787f29c4472 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.331017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56626ebb-462f-499f-a7ac-a787f29c4472 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.331283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56626ebb-462f-499f-a7ac-a787f29c4472 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.331909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56626ebb-462f-499f-a7ac-a787f29c4472 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:57.332244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56626ebb-462f-499f-a7ac-a787f29c4472 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:57.333568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56626ebb-462f-499f-a7ac-a787f29c4472 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:57.339313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 484/974] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:41:57 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:57.341193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c238b7b-79fe-4402-925f-67010e0c7b7f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.341460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c238b7b-79fe-4402-925f-67010e0c7b7f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.341735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c238b7b-79fe-4402-925f-67010e0c7b7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.354327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.361352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5c238b7b-79fe-4402-925f-67010e0c7b7f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:57.363043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5c238b7b-79fe-4402-925f-67010e0c7b7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:57.372635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 491/975] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:41:57 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:41:57.373560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.464045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 485/976] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:57.464045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c238b7b-79fe-4402-925f-67010e0c7b7f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.489524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5df8a-31fb-446e-a8e0-b48511d40fa2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:57.523107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.530581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:57.531180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.531454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.531718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.532454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] RBAC: Adding query filter params (interface=admin, service_id=72b004820f294b34be2258cee35b856d, region_id=RegionOne) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:57.532752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:57.534252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:57.544415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 492/977] 10.222.0.42 () {56 vars in 1249 bytes} [Fri Sep 16 01:41:57 2022] GET /identity/v3/endpoints?service_id=72b004820f294b34be2258cee35b856d&interface=admin®ion_id=RegionOne => generated 193 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:57.546150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.601357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c238b7b-79fe-4402-925f-67010e0c7b7f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:57.650462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1996788e-3a50-46ad-9150-17f9ecaa39b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:57.651719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1996788e-3a50-46ad-9150-17f9ecaa39b4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.652249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1996788e-3a50-46ad-9150-17f9ecaa39b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.653030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1996788e-3a50-46ad-9150-17f9ecaa39b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.657348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.663279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8a0888-d530-4543-bde8-b59381e7180d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:57.669795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1996788e-3a50-46ad-9150-17f9ecaa39b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:57.671277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1996788e-3a50-46ad-9150-17f9ecaa39b4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-1996788e-3a50-46ad-9150-17f9ecaa39b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:41:57.688323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:57.689305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:41:57.690183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:41:57.690183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:41:57.690183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:57.690183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 486/978] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:41:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:57.690971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2c3b1-dbd9-44ad-92f3-665e255e349d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.691073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2c3b1-dbd9-44ad-92f3-665e255e349d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.691330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff2c3b1-dbd9-44ad-92f3-665e255e349d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.692747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 487/979] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:57.697579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2c3b1-dbd9-44ad-92f3-665e255e349d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.706911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b868cfa-0acb-4fe1-9b86-eff0656d3463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:57.707510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b868cfa-0acb-4fe1-9b86-eff0656d3463 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.707787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b868cfa-0acb-4fe1-9b86-eff0656d3463 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.708085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b868cfa-0acb-4fe1-9b86-eff0656d3463 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.708805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b868cfa-0acb-4fe1-9b86-eff0656d3463 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:57.710158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b868cfa-0acb-4fe1-9b86-eff0656d3463 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:57.724047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 493/980] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:41:57 2022] GET /identity/v3/services => generated 1497 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:41:57.724047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387ab44a-131e-44bc-b172-df5b67000889 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.724047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387ab44a-131e-44bc-b172-df5b67000889 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.724047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387ab44a-131e-44bc-b172-df5b67000889 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.742015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-387ab44a-131e-44bc-b172-df5b67000889 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:57.743237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-387ab44a-131e-44bc-b172-df5b67000889 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:41:57.818566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.829716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff2c3b1-dbd9-44ad-92f3-665e255e349d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:57.832134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 494/981] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:57 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:41:57.836751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387ab44a-131e-44bc-b172-df5b67000889 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.873878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:57.874749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:57.875203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:57.875624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:57.876473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:57.876921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:57.878501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:57.898419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 488/982] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:41:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:57.899689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:57.961179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:57.965940 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387ab44a-131e-44bc-b172-df5b67000889 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:58.006375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:58.006907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:58.007185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:58.007516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:58.008190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:58.009614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:58.023433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ae0f79-0345-4662-a6ab-f423afecb28f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:58.024981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.sql.core [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 16 01:41:58.024981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 16 01:41:58.024981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [parameters: {'user_id': '116ddfd2c7014c47af4a8f559469010d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 16 01:41:58.024981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81424) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application conn.query(q) Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result.read() Sep 16 01:41:58.032711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 01:41:58.033599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application next(self.gen) Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application yield resource Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application next(self.gen) Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.session.rollback() Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.force_reraise() Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise self.value Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application session.commit() Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 16 01:41:58.034533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._prepare_impl() Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.session.flush() Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._flush(objects) Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application compat.raise_( Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application flush_context.execute() Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rec.execute(self) Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 16 01:41:58.035407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 16 01:41:58.036430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result = self._query(query) Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application conn.query(q) Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application result.read() Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 16 01:41:58.037416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [parameters: {'user_id': '116ddfd2c7014c47af4a8f559469010d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:58.038437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 16 01:41:58.039435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 16 01:41:58.040560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 16 01:41:58.040560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 16 01:41:58.040560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 16 01:41:58.040560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 16 01:41:58.040560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 16 01:41:58.040560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:58.040560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 495/983] 10.222.0.42 () {60 vars in 1131 bytes} [Fri Sep 16 01:41:57 2022] POST /identity/v3/users => generated 160 bytes in 202 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:58.048547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:58.063721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:58.064428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:58.064703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:58.064969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:58.072463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:58.074025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:41:58.086293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:58.087182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:41:58.088115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 489/984] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:41:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:58.093905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:58.119410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:58.173383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7140ff-91f3-4820-a268-85ae330474f7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:58.206319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc46f23-90a6-4a2d-987b-69edcaa04a8f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:58.208646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:58.209276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:58.209557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:58.209825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:58.216275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:58.217846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:58.219591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:41:58.232449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:41:58.236065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 496/985] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:41:58 2022] GET /identity/v3/users/cinder => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:41:58.237171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:58.251743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:58.252408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:58.252674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:58.252936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:58.253622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:58.253921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:58.255341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:58.273765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 490/986] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:41:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:41:58.273765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:58.346232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95333a9-35ab-4605-92a7-d78419a39f43 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:58.359246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:41:58.375536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acfb9952-11cd-4eee-a181-c04ebcdab98d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:58.383884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:58.384525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:58.384804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:58.385076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:58.385718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:58.387977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:58.407403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 497/987] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:41:58 2022] POST /identity/v3/services => generated 242 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:41:58.408585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:41:58.431509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcbb865-f691-4f5b-8b3f-e846bc16631b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:58.432163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbb865-f691-4f5b-8b3f-e846bc16631b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:58.432439 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbb865-f691-4f5b-8b3f-e846bc16631b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:58.432705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcbb865-f691-4f5b-8b3f-e846bc16631b None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:58.433692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbb865-f691-4f5b-8b3f-e846bc16631b None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:58.434141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbb865-f691-4f5b-8b3f-e846bc16631b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:58.436087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcbb865-f691-4f5b-8b3f-e846bc16631b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:58.478767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 491/988] 10.222.0.42 () {56 vars in 1101 bytes} [Fri Sep 16 01:41:58 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:58.538521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85adc1b3-5bde-4112-a296-8734c3480f59 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:41:58.580587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a7b390-34be-4986-aa18-a7d84337377e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:41:58.581229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a7b390-34be-4986-aa18-a7d84337377e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:58.581508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a7b390-34be-4986-aa18-a7d84337377e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:58.581778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a7b390-34be-4986-aa18-a7d84337377e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:58.582544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a7b390-34be-4986-aa18-a7d84337377e None admin] RBAC: Adding query filter params (role.id=18761cd58fe9422485f4b8e316845930, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=f58ec160611e446193b43582a8155284) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:41:58.582854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a7b390-34be-4986-aa18-a7d84337377e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:41:58.584404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a7b390-34be-4986-aa18-a7d84337377e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:41:58.596099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 498/989] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:41:58 2022] GET /identity/v3/role_assignments?user.id=f58ec160611e446193b43582a8155284&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=18761cd58fe9422485f4b8e316845930 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:41:59.968690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebfa91c-1ed4-4987-8996-709af4b041cd None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:59.969234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebfa91c-1ed4-4987-8996-709af4b041cd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:59.969234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebfa91c-1ed4-4987-8996-709af4b041cd None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:59.970723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 492/990] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:41:59.975753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a645be-de18-4d77-9307-463c061fc8a3 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:41:59.976171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a645be-de18-4d77-9307-463c061fc8a3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:41:59.976347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a645be-de18-4d77-9307-463c061fc8a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:41:59.995032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-87a645be-de18-4d77-9307-463c061fc8a3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:41:59.997285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-87a645be-de18-4d77-9307-463c061fc8a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:00.027273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:00.068300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 499/991] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:41:59 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:00.337345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebfa91c-1ed4-4987-8996-709af4b041cd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:00.361487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:00.442207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebfa91c-1ed4-4987-8996-709af4b041cd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:00.478703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f6c415-74d7-4e1d-af17-840533b5d474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:00.479324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f6c415-74d7-4e1d-af17-840533b5d474 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.479598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f6c415-74d7-4e1d-af17-840533b5d474 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.479864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f6c415-74d7-4e1d-af17-840533b5d474 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.480539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f6c415-74d7-4e1d-af17-840533b5d474 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:00.481947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f6c415-74d7-4e1d-af17-840533b5d474 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-41f6c415-74d7-4e1d-af17-840533b5d474 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:00.490839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:00.492324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 16 01:42:00.493181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:00.494417 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 493/992] 10.222.0.42 () {56 vars in 1126 bytes} [Fri Sep 16 01:42:00 2022] GET /identity/v3/services/object-store => generated 93 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:00.498887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a645be-de18-4d77-9307-463c061fc8a3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:00.602168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a645be-de18-4d77-9307-463c061fc8a3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:00.618184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2e4161-5d09-408c-b85e-975ed9d5fd7f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.618736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2e4161-5d09-408c-b85e-975ed9d5fd7f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.619193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2e4161-5d09-408c-b85e-975ed9d5fd7f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.620668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 494/993] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:42:00.626625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a742a18-8de1-4556-b506-b255edaea41f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.626905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a742a18-8de1-4556-b506-b255edaea41f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.627170 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a742a18-8de1-4556-b506-b255edaea41f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.639458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:00.640096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.640373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.640638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.641260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:00.641553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:00.642890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:00.647681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4a742a18-8de1-4556-b506-b255edaea41f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:00.648096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 500/994] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:42:00 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:00.648900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4a742a18-8de1-4556-b506-b255edaea41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:00.653442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:00.687971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:00.724044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 495/995] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:00 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:00.765578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03dc1c-20b3-40d8-96f6-3bb1b3fb25f3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:00.804531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b860c55-6836-4104-b6a8-fe10444de373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:00.805371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b860c55-6836-4104-b6a8-fe10444de373 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.805805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b860c55-6836-4104-b6a8-fe10444de373 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.806204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b860c55-6836-4104-b6a8-fe10444de373 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.807032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b860c55-6836-4104-b6a8-fe10444de373 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:00.807519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b860c55-6836-4104-b6a8-fe10444de373 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:00.810111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b860c55-6836-4104-b6a8-fe10444de373 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:00.815445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 501/996] 10.222.0.42 () {56 vars in 1122 bytes} [Fri Sep 16 01:42:00 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:00.820993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a742a18-8de1-4556-b506-b255edaea41f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:00.826699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1d709d-9580-4ad3-8042-1334e90840fc None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.826993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1d709d-9580-4ad3-8042-1334e90840fc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.827271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1d709d-9580-4ad3-8042-1334e90840fc None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.828642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 502/997] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:00.834209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490be5f9-ee12-46ed-9ca3-3a49ba1e24e4 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.834657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490be5f9-ee12-46ed-9ca3-3a49ba1e24e4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.835056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490be5f9-ee12-46ed-9ca3-3a49ba1e24e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.839890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:00.859165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-490be5f9-ee12-46ed-9ca3-3a49ba1e24e4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:00.860383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-490be5f9-ee12-46ed-9ca3-3a49ba1e24e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:00.931349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 503/998] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:00 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:00.931571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a742a18-8de1-4556-b506-b255edaea41f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:00.969104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:00.969731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:00.969961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:00.970223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:00.970917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:00.972312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:00.995991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 496/999] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:42:00 2022] POST /identity/v3/endpoints => generated 325 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:00.997921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490be5f9-ee12-46ed-9ca3-3a49ba1e24e4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.027007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:01.113235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490be5f9-ee12-46ed-9ca3-3a49ba1e24e4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.158515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.159212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.159496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.159774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] PATH_INFO: `/v3/services/compute` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.160439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.161920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:01.171765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:01.172965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:01.173800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 504/1000] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:42:00 2022] GET /identity/v3/services/compute => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:01.180480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.209468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.265674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:01.295417 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ef7f6e-554f-434b-9976-c7cc7acca50e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.303658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:01.322786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd6ddf1-c2b9-4a69-afbd-c90730d59e4b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.331809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.332431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.332711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.333246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.333616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] RBAC: Adding query filter params (name=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:01.333909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.335253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.340240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 497/1001] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:42:01 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:01.345859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.361184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.362090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.362563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.362840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] PATH_INFO: `/v3/services/image` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.363548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.364883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:01.374742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:01.375754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:01.377073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 505/1002] 10.222.0.42 () {56 vars in 1105 bytes} [Fri Sep 16 01:42:01 2022] GET /identity/v3/services/image => generated 86 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:01.381667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.455632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f57eaa-cdf1-469e-a5a8-558248cab73e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.488335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe6bb92-bc9b-44cd-b28f-58774dbde584 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.492254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.492856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.493137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.493405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.494030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:01.494395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.495757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.503140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 498/1003] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:42:01 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:01.507918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.512015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:01.527550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.528176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.528480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.528751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.529383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] RBAC: Adding query filter params (name=image) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:01.529678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.531015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.535966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 506/1004] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:01 2022] GET /identity/v3/services?name=image => generated 122 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:01.541486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.556141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:01.624309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99609be-ae21-4f35-afc8-30eaf2d27905 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.645726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8347c4-f4f4-47ec-abeb-0dfa7d812a7b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.660059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.660919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.661369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.661809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.662838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] RBAC: Adding query filter params (interface=public, service_id=d8d8c2a87ec74f0684690487a9e6190f, region_id=RegionOne) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:01.663368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.664914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.671280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 499/1005] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:42:01 2022] GET /identity/v3/endpoints?service_id=d8d8c2a87ec74f0684690487a9e6190f&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:01.676751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:01.684969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c02340-f921-4b1f-8409-e2dbb98c9f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.685584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c02340-f921-4b1f-8409-e2dbb98c9f4a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.685864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c02340-f921-4b1f-8409-e2dbb98c9f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.686244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c02340-f921-4b1f-8409-e2dbb98c9f4a None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.686880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c02340-f921-4b1f-8409-e2dbb98c9f4a None admin] RBAC: Adding query filter params (type=image) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:01.687180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c02340-f921-4b1f-8409-e2dbb98c9f4a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.688564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c02340-f921-4b1f-8409-e2dbb98c9f4a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.693035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 507/1006] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:01 2022] GET /identity/v3/services?type=image => generated 122 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:01.751032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:01.791030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ab9417-b9f0-4f8e-bf5a-ea3955e12033 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:01.834914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585b131e-f7ed-4442-90ea-71ac0a15ff41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:01.835529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585b131e-f7ed-4442-90ea-71ac0a15ff41 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:01.835800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585b131e-f7ed-4442-90ea-71ac0a15ff41 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:01.836094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585b131e-f7ed-4442-90ea-71ac0a15ff41 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:01.836692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585b131e-f7ed-4442-90ea-71ac0a15ff41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:01.838068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585b131e-f7ed-4442-90ea-71ac0a15ff41 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:01.851960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 500/1007] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:42:01 2022] GET /identity/v3/services => generated 1727 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:42:03.208026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3f0e3-65f3-40cc-b92d-d61a0c84c12e None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:03.208590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3f0e3-65f3-40cc-b92d-d61a0c84c12e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:03.208590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca3f0e3-65f3-40cc-b92d-d61a0c84c12e None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:03.210040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 508/1008] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:03.215262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ae55c-1f49-48de-bd57-22d3ad1f212b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:03.215675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ae55c-1f49-48de-bd57-22d3ad1f212b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:03.215872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33ae55c-1f49-48de-bd57-22d3ad1f212b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:03.234298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c33ae55c-1f49-48de-bd57-22d3ad1f212b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:03.235472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c33ae55c-1f49-48de-bd57-22d3ad1f212b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:03.302690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 501/1009] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:03 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:03.573442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3f0e3-65f3-40cc-b92d-d61a0c84c12e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:03.652742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:03.678050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca3f0e3-65f3-40cc-b92d-d61a0c84c12e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:03.712392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2727a929-15c4-4780-8178-4f33a20a74c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:03.713025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2727a929-15c4-4780-8178-4f33a20a74c5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:03.713319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2727a929-15c4-4780-8178-4f33a20a74c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:03.713576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2727a929-15c4-4780-8178-4f33a20a74c5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:03.714930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2727a929-15c4-4780-8178-4f33a20a74c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:03.716399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2727a929-15c4-4780-8178-4f33a20a74c5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:03.718133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 509/1010] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:03.723862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33ae55c-1f49-48de-bd57-22d3ad1f212b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:03.789617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:03.839313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33ae55c-1f49-48de-bd57-22d3ad1f212b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:03.861921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b7255-c320-4e8f-869b-533910cb7eb7 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:03.862204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b7255-c320-4e8f-869b-533910cb7eb7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:03.862482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b7255-c320-4e8f-869b-533910cb7eb7 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:03.863816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 510/1011] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:42:03.868899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b90f0f-57d9-4cfe-91cc-1ef0f8bcad86 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:03.869269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b90f0f-57d9-4cfe-91cc-1ef0f8bcad86 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:03.869608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b90f0f-57d9-4cfe-91cc-1ef0f8bcad86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:03.877643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c669f49-00bb-4a56-b328-bb8567ef373e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:03.878242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c669f49-00bb-4a56-b328-bb8567ef373e None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:03.878524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c669f49-00bb-4a56-b328-bb8567ef373e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:03.878796 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c669f49-00bb-4a56-b328-bb8567ef373e None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:03.879454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c669f49-00bb-4a56-b328-bb8567ef373e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:03.880920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c669f49-00bb-4a56-b328-bb8567ef373e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:03.889599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d4b90f0f-57d9-4cfe-91cc-1ef0f8bcad86 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:03.890928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d4b90f0f-57d9-4cfe-91cc-1ef0f8bcad86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:03.916022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 502/1012] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:42:03 2022] POST /identity/v3/projects => generated 311 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:03.969014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 511/1013] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:03 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:04.020797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb7c512-4c29-4395-811f-1d3d29eef8d9 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:04.021117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb7c512-4c29-4395-811f-1d3d29eef8d9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:04.021409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb7c512-4c29-4395-811f-1d3d29eef8d9 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:04.022855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 503/1014] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:04.029003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5081834-0308-4807-bc72-049bd2ed7e4d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:04.029298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5081834-0308-4807-bc72-049bd2ed7e4d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:04.029577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5081834-0308-4807-bc72-049bd2ed7e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:04.049528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f5081834-0308-4807-bc72-049bd2ed7e4d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:04.050750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f5081834-0308-4807-bc72-049bd2ed7e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:04.123776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 512/1015] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:04 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:04.242340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb7c512-4c29-4395-811f-1d3d29eef8d9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:04.347241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb7c512-4c29-4395-811f-1d3d29eef8d9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:04.361311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:04.384262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:04.384793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:04.385072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:04.385341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:04.386246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:04.388185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:04.392548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5081834-0308-4807-bc72-049bd2ed7e4d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:04.397369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:04.405135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 504/1016] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:42:04 2022] POST /identity/v3/services => generated 242 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:04.509296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5081834-0308-4807-bc72-049bd2ed7e4d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:04.553765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:04.554407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:04.554683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:04.554952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:04.555535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:04.556927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:04.566727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:04.567818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:04.569179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 513/1017] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:42:04 2022] GET /identity/v3/services/compute => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:04.574425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:04.678821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8416f84e-6a90-4b4a-9b29-70021f3f2c01 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:04.713578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:04.714179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:04.714483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:04.714753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:04.715388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] RBAC: Adding query filter params (name=compute) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:04.715687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:04.717106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:04.722285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 505/1018] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:42:04 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:04.728114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:04.792580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:04.842085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95132fea-4019-45bd-be1a-48127b1f6615 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:04.875841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec4660b-e0e7-43b9-8818-797fef02c9b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:04.876498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec4660b-e0e7-43b9-8818-797fef02c9b2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:04.876782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec4660b-e0e7-43b9-8818-797fef02c9b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:04.877050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec4660b-e0e7-43b9-8818-797fef02c9b2 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:04.877695 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec4660b-e0e7-43b9-8818-797fef02c9b2 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:04.877994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec4660b-e0e7-43b9-8818-797fef02c9b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:04.879399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec4660b-e0e7-43b9-8818-797fef02c9b2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:04.890181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 514/1019] 10.222.0.42 () {56 vars in 1112 bytes} [Fri Sep 16 01:42:04 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:04.891150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:04.934262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:04.997442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e436715-12f5-44bc-a343-5d723a8d8cfc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:05.031139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e002090-e7df-4e46-972b-5ff950ff24ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:05.031736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e002090-e7df-4e46-972b-5ff950ff24ba None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:05.032039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e002090-e7df-4e46-972b-5ff950ff24ba None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:05.032316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e002090-e7df-4e46-972b-5ff950ff24ba None admin] PATH_INFO: `/v3/endpoints` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:05.032968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e002090-e7df-4e46-972b-5ff950ff24ba None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:05.034400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e002090-e7df-4e46-972b-5ff950ff24ba None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:05.059971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 506/1020] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:42:04 2022] POST /identity/v3/endpoints => generated 334 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:42:06.170180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f1bc4-4739-460d-83e9-01ebf39a58c0 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:06.170702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f1bc4-4739-460d-83e9-01ebf39a58c0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:06.170781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f1bc4-4739-460d-83e9-01ebf39a58c0 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:06.172129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 515/1021] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:06.177595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e091b03b-e648-4466-9e44-b3a08b8199b6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:06.178020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e091b03b-e648-4466-9e44-b3a08b8199b6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:06.178165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e091b03b-e648-4466-9e44-b3a08b8199b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:06.197321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e091b03b-e648-4466-9e44-b3a08b8199b6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:06.198499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e091b03b-e648-4466-9e44-b3a08b8199b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:06.279000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 507/1022] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:06.550896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f1bc4-4739-460d-83e9-01ebf39a58c0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:06.602666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2aed45-93e6-45fd-a683-e2a14aaedae0 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:06.602998 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2aed45-93e6-45fd-a683-e2a14aaedae0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:06.603208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2aed45-93e6-45fd-a683-e2a14aaedae0 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:06.604587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 508/1023] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:06.609714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06321959-561e-422d-98a8-d0bec9857528 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:06.609995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06321959-561e-422d-98a8-d0bec9857528 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:06.610264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06321959-561e-422d-98a8-d0bec9857528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:06.628640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-06321959-561e-422d-98a8-d0bec9857528 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:06.629807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-06321959-561e-422d-98a8-d0bec9857528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:06.644823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:06.663005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f1bc4-4739-460d-83e9-01ebf39a58c0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:06.693067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:06.703978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:06.704178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 509/1024] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:06 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:06.704275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:06.704275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:06.704275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:06.708660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:06.710139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:06.711796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 516/1025] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:06.717882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06321959-561e-422d-98a8-d0bec9857528 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:06.827772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06321959-561e-422d-98a8-d0bec9857528 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:06.862659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:06.863260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:06.863547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:06.863798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:06.864477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:06.865894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:06.889624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 510/1026] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:42:06 2022] POST /identity/v3/users => generated 285 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:06.974218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:07.082574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee9a4f9-b9d3-4f65-ab90-2f1ee3420c88 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:07.118705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:07.119340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.119615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.119881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] PATH_INFO: `/v3/services/image` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.121108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:07.122858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:07.133194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:07.134299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:07.135203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 517/1027] 10.222.0.42 () {56 vars in 1105 bytes} [Fri Sep 16 01:42:06 2022] GET /identity/v3/services/image => generated 86 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:07.143253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:07.153938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:07.252613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689a3b4e-6c76-4087-a4d9-8dd8de7df7ae None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:07.290201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e13c597-5704-4415-908f-afc041d19d44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:07.290803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13c597-5704-4415-908f-afc041d19d44 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.291080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13c597-5704-4415-908f-afc041d19d44 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.291345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13c597-5704-4415-908f-afc041d19d44 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.291984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13c597-5704-4415-908f-afc041d19d44 None admin] RBAC: Adding query filter params (name=image) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:07.292288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13c597-5704-4415-908f-afc041d19d44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:07.293860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13c597-5704-4415-908f-afc041d19d44 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:07.299018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 511/1028] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:07 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:07.304511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:07.361156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b336d8-241d-48fe-8f6b-8762c848531d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.361442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b336d8-241d-48fe-8f6b-8762c848531d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.361711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b336d8-241d-48fe-8f6b-8762c848531d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.363048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 512/1029] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:42:07.368216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a755c5-a7d4-4f0d-a399-7c9a7fc6eb06 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.368505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a755c5-a7d4-4f0d-a399-7c9a7fc6eb06 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.368780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a755c5-a7d4-4f0d-a399-7c9a7fc6eb06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.370332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:07.389226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-70a755c5-a7d4-4f0d-a399-7c9a7fc6eb06 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:07.390580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-70a755c5-a7d4-4f0d-a399-7c9a7fc6eb06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:07.417459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adef55-ade9-4516-835b-93c3274416f2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:07.454432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:07.455035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.455312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.455579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.456232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] RBAC: Adding query filter params (type=image) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:07.456527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:07.457872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:07.461236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:07.467922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 518/1030] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:07 2022] GET /identity/v3/services?type=image => generated 350 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:07.469697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:07.472219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 513/1031] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:07 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:07.576180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073ab5cc-1402-44d3-82a3-60e32c627260 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:07.599531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:07.611763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:07.612581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.613037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.613461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.614307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] RBAC: Adding query filter params (interface=public, service_id=f1cc3c1497024d2d9df933289bdd1ae3, region_id=RegionOne) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:07.614758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:07.616289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:07.622408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 519/1032] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:42:07 2022] GET /identity/v3/endpoints?service_id=f1cc3c1497024d2d9df933289bdd1ae3&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:07.628538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a755c5-a7d4-4f0d-a399-7c9a7fc6eb06 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:07.732783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a755c5-a7d4-4f0d-a399-7c9a7fc6eb06 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:07.737344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:07.771910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:07.772526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.772969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.773235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.773825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:07.775986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:07.784923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 514/1033] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:42:07 2022] GET /identity/v3/services => generated 1957 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:42:07.852873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16ed3f3-b15b-47e4-91c9-90162f46b868 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:07.891041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:07.891600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:07.891879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:07.892178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:07.904893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:07.909870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:07.910825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:42:07.911661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:07.917868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 520/1034] 10.222.0.42 () {56 vars in 1120 bytes} [Fri Sep 16 01:42:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:07.918064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:07.986092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:08.031032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd02e844-65c4-4ec9-b65f-ffbf941235f9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:08.067337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:08.067953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:08.068240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:08.068515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:08.069166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:08.069465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:08.070823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:08.079178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 515/1035] 10.222.0.42 () {56 vars in 1115 bytes} [Fri Sep 16 01:42:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:08.085064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:08.135789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:08.189622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4028a301-c78f-420d-8397-6c6cbdd376ea None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:08.223419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:08.224073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:08.224352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:08.224617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:08.235810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:08.237266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:08.252020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:08.253017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:42:08.253903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:08.253903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:08.253903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:08.253903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 521/1036] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:08.259903 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:08.366159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49212a3d-6b09-42ae-afd2-c2ba2bf36810 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:08.392376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:08.401848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:08.402448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:08.402728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:08.402992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:08.403624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:08.403920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:08.405303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:08.422336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 516/1037] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:08.423838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:08.523198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:08.528347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85db384-6c07-47ba-9824-feebce7511ba None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:08.563534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:08.564191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:08.564472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:08.564740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:08.572405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:08.574154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:42:08.585239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:08.586183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:42:08.587120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:08.591853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 522/1038] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:42:08 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:08.593087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:08.697451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7bd903-eea1-490f-9a26-d4d0cbe2c4f5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:08.733763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:08.734361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:08.734630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:08.734900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:08.735617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:08.735919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:08.737361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:08.759695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 517/1039] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:42:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:08.761830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:08.869512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4f0695-decd-4a3d-bbaf-9729a41ff2d7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:08.904539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d776542a-7d19-4391-969f-c1603080d1b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:08.905139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d776542a-7d19-4391-969f-c1603080d1b1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:08.905416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d776542a-7d19-4391-969f-c1603080d1b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:08.905684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d776542a-7d19-4391-969f-c1603080d1b1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:08.933921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:08.977077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d776542a-7d19-4391-969f-c1603080d1b1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:08.978518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d776542a-7d19-4391-969f-c1603080d1b1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:09.082324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 523/1040] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:08 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:09.087814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.134499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:09.150648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8f7a52-0f39-4a5b-84cc-56e99a7b0636 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.150934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8f7a52-0f39-4a5b-84cc-56e99a7b0636 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.151206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c8f7a52-0f39-4a5b-84cc-56e99a7b0636 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.153703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 524/1041] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:09.158500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7548b7-d987-48f2-b682-2a67b056fee7 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.158781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7548b7-d987-48f2-b682-2a67b056fee7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.159052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7548b7-d987-48f2-b682-2a67b056fee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.177655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7c7548b7-d987-48f2-b682-2a67b056fee7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:09.178820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7c7548b7-d987-48f2-b682-2a67b056fee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:09.193535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed6c583-8bb8-447c-81c3-c121fdb88dbd None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.229198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:09.229810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.230275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.230722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.237998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:09.241078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:09.242506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:09.251484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 525/1042] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:09 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:09.258114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:09.259053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:42:09.261843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:09.261843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:09.261843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:09.261843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 518/1043] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:09.267924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7548b7-d987-48f2-b682-2a67b056fee7 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.364622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7548b7-d987-48f2-b682-2a67b056fee7 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.399418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:09.400059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.400337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.400603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.401234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:09.401531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:09.402924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:09.414297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 526/1044] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:09.419879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.514288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:09.521374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079b2404-b19a-48b0-9b3f-bbde9abb006e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.523810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.558725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:09.559337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.559612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.559878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.566749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:09.568278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:09.576280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:42:09.580616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:09.581552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:09.582442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 519/1045] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:09.587371 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.637988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6808dbdc-4c28-449f-b8d6-ceee2a6129a1 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.673468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:09.674092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.674369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.674635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.685886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:09.687981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84a6edf-7df5-45df-acb3-627c97b68fa2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:09.690726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:09.691860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:09.693060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 527/1046] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:09.699352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.729585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:09.730706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.730973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.731243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.731951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:09.732267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:09.733733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:09.750813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 520/1047] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:09.752168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.771111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:09.817591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06a8c70a-e584-4069-8731-cacea2b9ff1b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.830863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:09.858079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:09.858778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.859126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.859467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.860224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:09.860589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:09.860701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1fd6a1-0233-473d-875b-557869597eb6 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.862101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:09.872603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 528/1048] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:09.878576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:09.905173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945e2fd9-2b5c-4e40-b9a5-94f59fa07e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:09.905695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e2fd9-2b5c-4e40-b9a5-94f59fa07e85 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:09.905974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e2fd9-2b5c-4e40-b9a5-94f59fa07e85 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:09.906244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945e2fd9-2b5c-4e40-b9a5-94f59fa07e85 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:09.906963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945e2fd9-2b5c-4e40-b9a5-94f59fa07e85 None admin] RBAC: Adding query filter params (role.id=18761cd58fe9422485f4b8e316845930, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=603c81d781754201a5ac5b7d9b84fab6) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:09.907270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945e2fd9-2b5c-4e40-b9a5-94f59fa07e85 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:09.908738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945e2fd9-2b5c-4e40-b9a5-94f59fa07e85 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:09.914666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 521/1049] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/role_assignments?user.id=603c81d781754201a5ac5b7d9b84fab6&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=18761cd58fe9422485f4b8e316845930 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:09.992344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f499a8f-8c86-4e8b-ad23-7561a5ac495f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:09.999624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b1419f-7582-4fe4-8fbc-dcb65654696f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.000075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b1419f-7582-4fe4-8fbc-dcb65654696f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.000188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b1419f-7582-4fe4-8fbc-dcb65654696f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.004402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 522/1050] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:42:10.007094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0d2820-fbbe-4f30-aaf3-65eb13113285 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.007373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0d2820-fbbe-4f30-aaf3-65eb13113285 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.007641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0d2820-fbbe-4f30-aaf3-65eb13113285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.028601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3c0d2820-fbbe-4f30-aaf3-65eb13113285 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:10.029826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3c0d2820-fbbe-4f30-aaf3-65eb13113285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:10.036403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:10.037056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.037334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.037603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] PATH_INFO: `/v3/users/bd95ab25ece2434798bf14f92ab21d96` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.056046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] RBAC: Authorizing `identity:get_user(user_id=bd95ab25ece2434798bf14f92ab21d96)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:10.057500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:10.059802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 529/1051] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:09 2022] GET /identity/v3/users/bd95ab25ece2434798bf14f92ab21d96 => generated 285 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:10.065452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:10.099207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:10.114132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 523/1052] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:10 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:10.172009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619ca2c4-9094-42c9-8586-7883d9e0c123 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:10.207224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:10.207826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.208135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.208405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.277980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:10.279414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:10.308682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:10.383430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 530/1053] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:10 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 16 01:42:10.385030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0d2820-fbbe-4f30-aaf3-65eb13113285 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:10.387721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:10.398452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:10.492472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0d2820-fbbe-4f30-aaf3-65eb13113285 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:10.496140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93171d00-9aa7-4a82-8d04-0e9bedc65a12 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:10.527400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:10.528909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:10.529891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.530208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.530642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.531921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] RBAC: Authorizing `identity:get_project(project_id=6dc938ea232a4dcfb301d076660f3d5e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:10.533441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:10.535063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 524/1054] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:10 2022] GET /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:10.536016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:10.536836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.537309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.537802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] PATH_INFO: `/v3/services/image` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.538567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:10.540162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:10.540898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:10.550535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:10.551529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 16 01:42:10.552474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:10.559603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 531/1055] 10.222.0.42 () {56 vars in 1105 bytes} [Fri Sep 16 01:42:10 2022] GET /identity/v3/services/image => generated 86 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:10.561043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:10.628612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:10.650951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d60cc5-95cc-4aa9-a27e-e16b1233abf5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:10.675585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3054c05-e8dc-44f8-ae49-c90d3703c51c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:10.689067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:10.689659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.689942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.690205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.690922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=6dc938ea232a4dcfb301d076660f3d5e, user.id=bd95ab25ece2434798bf14f92ab21d96) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:10.691226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:10.692682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:10.698228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 525/1056] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:10 2022] GET /identity/v3/role_assignments?user.id=bd95ab25ece2434798bf14f92ab21d96&scope.project.id=6dc938ea232a4dcfb301d076660f3d5e&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:10.711629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:10.712624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.713099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.713573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.714460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] RBAC: Adding query filter params (name=image) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:10.714957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:10.716653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:10.722545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 532/1057] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:10 2022] GET /identity/v3/services?name=image => generated 122 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:10.728714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:10.851335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc3ea10-f314-42d7-acc3-091351dbfc5f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:10.890433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa104c73-19e7-4f52-ac98-8227f77ed283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:10.891037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa104c73-19e7-4f52-ac98-8227f77ed283 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:10.891316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa104c73-19e7-4f52-ac98-8227f77ed283 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:10.891584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa104c73-19e7-4f52-ac98-8227f77ed283 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:10.892298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa104c73-19e7-4f52-ac98-8227f77ed283 None admin] RBAC: Adding query filter params (type=image) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:10.892604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa104c73-19e7-4f52-ac98-8227f77ed283 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:10.893968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa104c73-19e7-4f52-ac98-8227f77ed283 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:10.904061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 526/1058] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:10 2022] GET /identity/v3/services?type=image => generated 350 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:10.905603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:10.966240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:11.010534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ab8c95-b838-4428-b8fa-bd3daac213dc None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:11.043656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66404f20-9eb9-4703-968a-111879ff9d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:11.044348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66404f20-9eb9-4703-968a-111879ff9d20 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:11.044621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66404f20-9eb9-4703-968a-111879ff9d20 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:11.044884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66404f20-9eb9-4703-968a-111879ff9d20 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:11.045549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66404f20-9eb9-4703-968a-111879ff9d20 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:11.046885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66404f20-9eb9-4703-968a-111879ff9d20 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:11.076058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 533/1059] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:42:10 2022] POST /identity/v3/endpoints => generated 327 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:42:12.099344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519fee3-1f94-447b-b34b-eeb02623e638 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:12.099866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519fee3-1f94-447b-b34b-eeb02623e638 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:12.099866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519fee3-1f94-447b-b34b-eeb02623e638 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:12.101131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 527/1060] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:12.106536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcba862-fa1f-46f4-8b1d-9b6d6d795128 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:12.106981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcba862-fa1f-46f4-8b1d-9b6d6d795128 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:12.107062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcba862-fa1f-46f4-8b1d-9b6d6d795128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:12.125965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0bcba862-fa1f-46f4-8b1d-9b6d6d795128 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:12.127142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0bcba862-fa1f-46f4-8b1d-9b6d6d795128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:12.201816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 534/1061] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:12.471924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519fee3-1f94-447b-b34b-eeb02623e638 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:12.481916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:12.578040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519fee3-1f94-447b-b34b-eeb02623e638 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:12.611561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:12.612263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:12.612535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:12.612798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:12.624060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:12.628802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:12.629717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:42:12.630568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:12.635058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 528/1062] 10.222.0.42 () {56 vars in 1120 bytes} [Fri Sep 16 01:42:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:12.636793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcba862-fa1f-46f4-8b1d-9b6d6d795128 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:12.678352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:12.738087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcba862-fa1f-46f4-8b1d-9b6d6d795128 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:12.776829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fada3a-7c04-4512-a8a2-7b98d81e35db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:12.777892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fada3a-7c04-4512-a8a2-7b98d81e35db None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:12.778282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fada3a-7c04-4512-a8a2-7b98d81e35db None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:12.778642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fada3a-7c04-4512-a8a2-7b98d81e35db None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:12.779406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fada3a-7c04-4512-a8a2-7b98d81e35db None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:12.779799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fada3a-7c04-4512-a8a2-7b98d81e35db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:12.781271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fada3a-7c04-4512-a8a2-7b98d81e35db None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:12.789931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 535/1063] 10.222.0.42 () {56 vars in 1115 bytes} [Fri Sep 16 01:42:12 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:12.798734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:12.865985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b52af5-80de-4866-a0fd-76e089bb10c2 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:12.866266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b52af5-80de-4866-a0fd-76e089bb10c2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:12.866617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b52af5-80de-4866-a0fd-76e089bb10c2 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:12.868001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 536/1064] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:42:12.873004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:12.873151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992a7cfe-e8d4-4b19-b22e-cd5536286307 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:12.873347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992a7cfe-e8d4-4b19-b22e-cd5536286307 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:12.873620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992a7cfe-e8d4-4b19-b22e-cd5536286307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:12.893558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-992a7cfe-e8d4-4b19-b22e-cd5536286307 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:12.894751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-992a7cfe-e8d4-4b19-b22e-cd5536286307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:12.913044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaea007-26d2-4212-8f49-85ed6c43e728 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:12.949697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:12.950311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:12.950585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:12.950852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:12.961441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:12.962862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:12.965737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 537/1065] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:12.977480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:12.978432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:42:12.979332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:12.979332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:12.979332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:12.984157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 529/1066] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:12.984961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992a7cfe-e8d4-4b19-b22e-cd5536286307 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.017923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:13.090443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992a7cfe-e8d4-4b19-b22e-cd5536286307 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.124239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.124877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.125131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.125403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.126064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:13.126368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.127773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.145728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 538/1067] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:42:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:13.147342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.238115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.256403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9879509a-0626-489d-b88b-a4905ceee3f5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.273615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:13.291018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c18a29-bd0b-4166-816f-c404ed1774e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.291637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c18a29-bd0b-4166-816f-c404ed1774e9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.291912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c18a29-bd0b-4166-816f-c404ed1774e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.292845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c18a29-bd0b-4166-816f-c404ed1774e9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.300353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c18a29-bd0b-4166-816f-c404ed1774e9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.301779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c18a29-bd0b-4166-816f-c404ed1774e9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d1c18a29-bd0b-4166-816f-c404ed1774e9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:42:13.313228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:13.314211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:13.315144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 530/1068] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/users/nova => generated 82 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:13.315723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebe11cd-7d15-4325-8c58-a4333c8730c9 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.315842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebe11cd-7d15-4325-8c58-a4333c8730c9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.316130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebe11cd-7d15-4325-8c58-a4333c8730c9 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.317617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 531/1069] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:13.319825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebe11cd-7d15-4325-8c58-a4333c8730c9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.333775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:13.344033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebd87ac-225c-4d9b-abbe-22464a2daa37 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.377900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f61f4-23f2-4de0-a3a2-a79e21bbbac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.378513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f61f4-23f2-4de0-a3a2-a79e21bbbac3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.378788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f61f4-23f2-4de0-a3a2-a79e21bbbac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.379054 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f61f4-23f2-4de0-a3a2-a79e21bbbac3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.390377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549f61f4-23f2-4de0-a3a2-a79e21bbbac3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-549f61f4-23f2-4de0-a3a2-a79e21bbbac3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:13.395099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:13.400056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 539/1070] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:13.400963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc980ce0-0f35-48b4-a46d-f1c9876eebcf None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.401791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc980ce0-0f35-48b4-a46d-f1c9876eebcf None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.401791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc980ce0-0f35-48b4-a46d-f1c9876eebcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.417992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-fc980ce0-0f35-48b4-a46d-f1c9876eebcf None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:13.419294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fc980ce0-0f35-48b4-a46d-f1c9876eebcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:13.426614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebe11cd-7d15-4325-8c58-a4333c8730c9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.460435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.461021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.461300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.461572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.462284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:13.462583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.464054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.481512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 532/1071] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:13.482301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.487649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 540/1072] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:13.488946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc980ce0-0f35-48b4-a46d-f1c9876eebcf None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.522948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:13.581636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:13.593663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86d66ee-30e4-4a63-87be-05736490a9ac None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.600016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc980ce0-0f35-48b4-a46d-f1c9876eebcf None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.628620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.629271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.629544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.629812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.630460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:13.630753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.632128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.638353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.639047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.639395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.639730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.647017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 533/1073] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:13.647755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.650304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.652339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:13.667259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:13.668832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:42:13.669764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:13.669764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:13.669764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:13.669764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 541/1074] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:13.674103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.754742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:13.758624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790a3598-6d6a-4a5d-b23a-03f2d0cbb052 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.779777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad0de03-0a0a-4e27-bc22-4d561ca8e2a7 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.794778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.795396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.795672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.796025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] PATH_INFO: `/v3/users/bd95ab25ece2434798bf14f92ab21d96` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.797106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] RBAC: Authorizing `identity:get_user(user_id=bd95ab25ece2434798bf14f92ab21d96)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.798532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.804831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 534/1075] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/users/bd95ab25ece2434798bf14f92ab21d96 => generated 285 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:13.804831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.817125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.817732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.818010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.818277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.818910 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:13.819210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.820591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.822005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:13.833625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 542/1076] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:13.834395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.907796 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb0b85-745c-455f-afba-bd8954f36209 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.934305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e1f29f-9f10-49e8-a3fb-8c8ac18a53ae None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:13.941204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.941801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.942075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.942341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.968019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:13.968639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:13.968915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:13.969184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:13.970281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] RBAC: Authorizing `identity:get_project(project_id=6dc938ea232a4dcfb301d076660f3d5e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:13.971693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:13.973347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 543/1077] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:13.974098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:13.981159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:14.016022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:14.017559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:14.033748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:14.074923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ac16e9-ad62-4952-9139-856d06c6af26 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:14.109097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:14.109725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:14.110002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:14.110270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:14.116335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 535/1078] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:14.116759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:14.117147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:14.118247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:42:14.131299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:14.132827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:14.133722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 544/1079] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:42:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:14.134436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:14.181656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:14.217062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7096377-ba23-4c3f-9a64-678af9bbd7c5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:14.237832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ac0bd-9aac-43a3-8fe7-a32c96ba872b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:14.252885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:14.253529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:14.253815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:14.254083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e/users/bd95ab25ece2434798bf14f92ab21d96/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:14.256028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] RBAC: Authorizing `identity:create_grant(project_id=6dc938ea232a4dcfb301d076660f3d5e, user_id=bd95ab25ece2434798bf14f92ab21d96, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:14.257459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:14.260060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:14.271028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 536/1080] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:42:14 2022] PUT /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e/users/bd95ab25ece2434798bf14f92ab21d96/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:42:14.271824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:14.272972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:14.273620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:14.273897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:14.274170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:14.291491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:14.293542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:14.310968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:14.312150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:42:14.313623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:14.313623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:14.313623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:14.313623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 545/1081] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:14.318420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:14.400396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3a305d-df9f-4385-a252-6eed11976f46 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:14.430895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:14.438794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1427a2-1612-450d-bcc8-1b2ff1d47b45 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:14.443518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:14.443518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:14.443518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:14.443518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:14.444843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:14.444843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:14.445709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:14.459251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 537/1082] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:42:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:14.466776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:14.485757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1cc128-57b6-4028-87ca-ab11897717a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:14.486359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cc128-57b6-4028-87ca-ab11897717a4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:14.486635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cc128-57b6-4028-87ca-ab11897717a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:14.486892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cc128-57b6-4028-87ca-ab11897717a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:14.487522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1cc128-57b6-4028-87ca-ab11897717a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:14.487814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1cc128-57b6-4028-87ca-ab11897717a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:14.489194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1cc128-57b6-4028-87ca-ab11897717a4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:14.502102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 546/1083] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:14.579582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc910cd-49b1-446a-ba28-16525d605200 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:14.616529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a34574-dcd2-41a4-9efb-bebd1ae46498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:14.617246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a34574-dcd2-41a4-9efb-bebd1ae46498 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:14.617529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a34574-dcd2-41a4-9efb-bebd1ae46498 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:14.617797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a34574-dcd2-41a4-9efb-bebd1ae46498 None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/603c81d781754201a5ac5b7d9b84fab6/roles/18761cd58fe9422485f4b8e316845930` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:14.620056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a34574-dcd2-41a4-9efb-bebd1ae46498 None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=603c81d781754201a5ac5b7d9b84fab6, role_id=18761cd58fe9422485f4b8e316845930)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:14.622154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a34574-dcd2-41a4-9efb-bebd1ae46498 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:14.639348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 538/1084] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:42:14 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/603c81d781754201a5ac5b7d9b84fab6/roles/18761cd58fe9422485f4b8e316845930 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:42:16.447723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832607b0-8d1e-4f4c-a24a-81a69151a773 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:16.448925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832607b0-8d1e-4f4c-a24a-81a69151a773 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:16.448925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832607b0-8d1e-4f4c-a24a-81a69151a773 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:16.450209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 547/1085] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:16.455628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11db9d9-f509-40a3-a9b9-821ca1794b13 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:16.456044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11db9d9-f509-40a3-a9b9-821ca1794b13 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:16.456214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11db9d9-f509-40a3-a9b9-821ca1794b13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:16.474512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a11db9d9-f509-40a3-a9b9-821ca1794b13 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:16.475679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a11db9d9-f509-40a3-a9b9-821ca1794b13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:16.517576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:16.543772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 539/1086] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:16.811983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832607b0-8d1e-4f4c-a24a-81a69151a773 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:16.847077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:16.855997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863663f-2312-4821-a3ed-b056fff3fbad None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:16.856286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863663f-2312-4821-a3ed-b056fff3fbad None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:16.856557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863663f-2312-4821-a3ed-b056fff3fbad None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:16.857888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 540/1087] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:42:16.862780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e00bf2-8de2-4904-847f-0c86cb32e899 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:16.863057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e00bf2-8de2-4904-847f-0c86cb32e899 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:16.863330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e00bf2-8de2-4904-847f-0c86cb32e899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:16.881970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e2e00bf2-8de2-4904-847f-0c86cb32e899 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:16.883153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e2e00bf2-8de2-4904-847f-0c86cb32e899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:16.914354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832607b0-8d1e-4f4c-a24a-81a69151a773 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:16.948073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:16.949299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:16.949580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:16.949846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:16.951379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 541/1088] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:16.960392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:16.965156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:16.966114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:16.966943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 548/1089] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:42:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:16.972254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e00bf2-8de2-4904-847f-0c86cb32e899 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.069740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e00bf2-8de2-4904-847f-0c86cb32e899 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.077466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.103497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.104115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.104395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.104687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.105334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:17.105631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.106981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.114698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 542/1090] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:42:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:17.121164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.211711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.218863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8630b399-8ad4-46ab-bb61-49397e97852d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.220116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.254368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.254995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.255274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.255551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] PATH_INFO: `/v3/users/bd95ab25ece2434798bf14f92ab21d96` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.256685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] RBAC: Authorizing `identity:get_user(user_id=bd95ab25ece2434798bf14f92ab21d96)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.258106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.259682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 549/1091] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/users/bd95ab25ece2434798bf14f92ab21d96 => generated 285 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:17.265584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.322263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101687e9-99a4-460f-9d6d-320985b79b6b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.356965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.357582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.357888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.358120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.368021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:17.369482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345ff68b-2399-46b4-8219-62ef3798b11b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:17.373766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:17.374867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:17.375713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 543/1092] 10.222.0.42 () {56 vars in 1120 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:17.380761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.410030 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.410702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.410981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.411255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.415757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.441573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.479813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.481300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.487054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9b1c4f-c2ba-41d7-92d3-aa29a0e7f4a1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.520700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.521298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.521573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.521838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.522486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:17.522786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.524156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.532075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 544/1093] 10.222.0.42 () {56 vars in 1115 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:17.537819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.578596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 550/1094] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 16 01:42:17.584409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.628283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.639127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018fdb7d-ad2e-4271-8dad-30f20bf6c5c8 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.644523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.672795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.673410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.673685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.673999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.684798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4291ee24-b66e-42a9-bbcc-e423e7caef09 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.685252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.686681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:17.701358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:17.702315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:42:17.703206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:17.703206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:17.703206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:17.704076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 545/1095] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:17.708199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.719541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.720192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.720493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.720759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.721852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] RBAC: Authorizing `identity:get_project(project_id=6dc938ea232a4dcfb301d076660f3d5e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.723241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.724874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 551/1096] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:17.731675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.816276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91713a74-9124-4c2e-a3de-262e19a8c7b9 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.829437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde19360-4384-418f-af33-0658193ff0ed None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:17.850389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.850984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.851261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.851530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.852195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:17.852670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.854039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.858568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.864839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:17.865428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:17.865701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:17.865964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:17.866333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 546/1097] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:17.868133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=6dc938ea232a4dcfb301d076660f3d5e, user.id=bd95ab25ece2434798bf14f92ab21d96) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:17.868133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:17.868392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:17.872190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:17.880730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 552/1098] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/role_assignments?user.id=bd95ab25ece2434798bf14f92ab21d96&scope.project.id=6dc938ea232a4dcfb301d076660f3d5e&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:17.881152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:17.979544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fbbe7a-0b2e-4c5d-b703-b9d25d3b2d8b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:18.015236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:18.015875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:18.016181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:18.016450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:18.024275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:18.025739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:42:18.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:18.040659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:18.041583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 547/1099] 10.222.0.42 () {56 vars in 1093 bytes} [Fri Sep 16 01:42:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:42:18.047186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:18.147028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1ed1e-a4b6-417f-bcef-859438bd23c0 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:18.179667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:18.180287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:18.180561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:18.180823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:18.181532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:18.181826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:18.183985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:18.200854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 553/1100] 10.222.0.42 () {56 vars in 1133 bytes} [Fri Sep 16 01:42:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:18.206477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:18.266715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:18.306546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beb9bf9-ba46-44ec-afaa-4848d5891408 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:18.339361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:18.340227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:18.340504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:18.340767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:18.407556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:18.408997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:18.466967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:18.505723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 548/1101] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:18.511869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:18.563865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:18.610094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9add92a6-34fc-4fee-a1df-55933010a442 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:18.644404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:18.645021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:18.645295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:18.645560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:18.656022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:18.657441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:18.671475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:18.672575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:42:18.673810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:18.673810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:42:18.673810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:18.673810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 554/1102] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:18.678382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:18.774151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd9df8-de6d-4b19-b217-d8dffc475581 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:18.814922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:18.815520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:18.815791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:18.816083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:18.816715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:18.817008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:18.818348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:18.830083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 549/1103] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:42:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:18.835604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:18.932375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b2adbb-75be-4482-bff5-54faa88ec532 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:18.935261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:18.965881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:18.966494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:18.966770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:18.967035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:18.973456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:18.974868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:42:18.986390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:18.987248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:18.988335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 555/1104] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:42:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:18.993229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:19.018378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:19.093248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820856f-1de2-4915-a87a-190d7f8b793b None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:19.126151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8244cc9f-0767-4c35-9b6b-6414b22b7d80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:19.126752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8244cc9f-0767-4c35-9b6b-6414b22b7d80 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:19.127031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8244cc9f-0767-4c35-9b6b-6414b22b7d80 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:19.127300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8244cc9f-0767-4c35-9b6b-6414b22b7d80 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:19.128010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8244cc9f-0767-4c35-9b6b-6414b22b7d80 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:19.128316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8244cc9f-0767-4c35-9b6b-6414b22b7d80 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:19.129743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8244cc9f-0767-4c35-9b6b-6414b22b7d80 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:19.142115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 550/1105] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:42:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:19.148112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:19.250015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533d752b-a545-4221-af5f-3b8480a89319 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:19.282660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df673f1-846a-4ad8-8e40-e7069a955d9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:19.283252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df673f1-846a-4ad8-8e40-e7069a955d9b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:19.283530 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df673f1-846a-4ad8-8e40-e7069a955d9b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:19.283800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df673f1-846a-4ad8-8e40-e7069a955d9b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:19.284548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df673f1-846a-4ad8-8e40-e7069a955d9b None admin] RBAC: Adding query filter params (role.id=18761cd58fe9422485f4b8e316845930, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=603c81d781754201a5ac5b7d9b84fab6) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:19.284856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df673f1-846a-4ad8-8e40-e7069a955d9b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:19.286272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df673f1-846a-4ad8-8e40-e7069a955d9b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:19.292111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 556/1106] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:19 2022] GET /identity/v3/role_assignments?user.id=603c81d781754201a5ac5b7d9b84fab6&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=18761cd58fe9422485f4b8e316845930 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:42:20.074629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fe9efa-4ecd-4994-bd76-4ef3c44e1d51 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:20.075126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fe9efa-4ecd-4994-bd76-4ef3c44e1d51 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:20.075126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fe9efa-4ecd-4994-bd76-4ef3c44e1d51 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:20.076435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 551/1107] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:20.081757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774a9710-37f9-4cfd-8ffd-f072e6d9dd3a None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:20.082198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774a9710-37f9-4cfd-8ffd-f072e6d9dd3a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:20.082291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774a9710-37f9-4cfd-8ffd-f072e6d9dd3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:20.101639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-774a9710-37f9-4cfd-8ffd-f072e6d9dd3a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:20.102808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-774a9710-37f9-4cfd-8ffd-f072e6d9dd3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:20.126435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:20.171722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 557/1108] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:20.446659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fe9efa-4ecd-4994-bd76-4ef3c44e1d51 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:20.528537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:20.546974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fe9efa-4ecd-4994-bd76-4ef3c44e1d51 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:20.580580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96bedd41-6738-4e17-a741-d6a12429d17b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:20.581187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bedd41-6738-4e17-a741-d6a12429d17b None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:20.581459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bedd41-6738-4e17-a741-d6a12429d17b None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:20.581722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bedd41-6738-4e17-a741-d6a12429d17b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:20.582795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bedd41-6738-4e17-a741-d6a12429d17b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:20.584223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96bedd41-6738-4e17-a741-d6a12429d17b None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:20.585733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 552/1109] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:20.592196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774a9710-37f9-4cfd-8ffd-f072e6d9dd3a None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:20.688354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774a9710-37f9-4cfd-8ffd-f072e6d9dd3a None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:20.720709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738e3b6e-fd3d-47e1-b059-cc4f46f0a020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:20.721235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e3b6e-fd3d-47e1-b059-cc4f46f0a020 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:20.721508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e3b6e-fd3d-47e1-b059-cc4f46f0a020 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:20.721774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e3b6e-fd3d-47e1-b059-cc4f46f0a020 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:20.722405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738e3b6e-fd3d-47e1-b059-cc4f46f0a020 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:20.723807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738e3b6e-fd3d-47e1-b059-cc4f46f0a020 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:20.743925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 558/1110] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:42:20 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:22.917751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42afba57-b076-49b3-9aeb-d0aa39eec6ff None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:22.918268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42afba57-b076-49b3-9aeb-d0aa39eec6ff None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:22.918268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42afba57-b076-49b3-9aeb-d0aa39eec6ff None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:22.919595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 553/1111] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:22.924740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96456cf-6648-4928-8804-47f0a645ed83 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:22.925123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96456cf-6648-4928-8804-47f0a645ed83 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:22.925296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96456cf-6648-4928-8804-47f0a645ed83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:22.944399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a96456cf-6648-4928-8804-47f0a645ed83 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:22.945569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a96456cf-6648-4928-8804-47f0a645ed83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:22.955063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:23.014156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 559/1112] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:23.287846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42afba57-b076-49b3-9aeb-d0aa39eec6ff None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:23.386671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42afba57-b076-49b3-9aeb-d0aa39eec6ff None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:23.420407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:23.421017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:23.421289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:23.421552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] PATH_INFO: `/v3/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:23.422931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] RBAC: Authorizing `identity:get_role(role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:23.424305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:23.425593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 554/1113] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:23 2022] GET /identity/v3/roles/9d68999390534c65b1cfc53d421f6412 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:23.431366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96456cf-6648-4928-8804-47f0a645ed83 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:23.527794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96456cf-6648-4928-8804-47f0a645ed83 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:23.560474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:23.561026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:23.561300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:23.561562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] PATH_INFO: `/v3/users/e6c64ffa43e54400b9b43123b91cf6e5` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:23.574139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] RBAC: Authorizing `identity:get_user(user_id=e6c64ffa43e54400b9b43123b91cf6e5)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:23.575545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:23.577129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 560/1114] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:23 2022] GET /identity/v3/users/e6c64ffa43e54400b9b43123b91cf6e5 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:23.582758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:23.601574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:23.679730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2331b7da-c366-421b-80ca-4254ba8ccd20 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:23.712808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:23.713414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:23.713751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:23.714019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:23.778640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:23.780099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:23.798374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:23.877250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 555/1115] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:23.883136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:23.886717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:23.979731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5989ff80-e20c-4fe5-beda-a94db276e895 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:24.011911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d95fe07-064a-450c-844c-eb17702346d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:24.012541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d95fe07-064a-450c-844c-eb17702346d2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:24.012813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d95fe07-064a-450c-844c-eb17702346d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:24.013077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d95fe07-064a-450c-844c-eb17702346d2 None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:24.014160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d95fe07-064a-450c-844c-eb17702346d2 None admin] RBAC: Authorizing `identity:get_project(project_id=6dc938ea232a4dcfb301d076660f3d5e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:24.015549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d95fe07-064a-450c-844c-eb17702346d2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:24.017150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 561/1116] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:23 2022] GET /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:24.023173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:24.118815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fa3918-6438-41ee-b2b1-f56f1118c331 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:24.136636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:24.154150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9c7358-b30c-403f-a8ba-d62cf32e24b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:24.154738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9c7358-b30c-403f-a8ba-d62cf32e24b4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:24.155014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9c7358-b30c-403f-a8ba-d62cf32e24b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:24.155280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9c7358-b30c-403f-a8ba-d62cf32e24b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:24.156028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9c7358-b30c-403f-a8ba-d62cf32e24b4 None admin] RBAC: Adding query filter params (role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=6dc938ea232a4dcfb301d076660f3d5e, user.id=e6c64ffa43e54400b9b43123b91cf6e5) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:24.156329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9c7358-b30c-403f-a8ba-d62cf32e24b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:24.157737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9c7358-b30c-403f-a8ba-d62cf32e24b4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:24.162978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 556/1117] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:24 2022] GET /identity/v3/role_assignments?user.id=e6c64ffa43e54400b9b43123b91cf6e5&scope.project.id=6dc938ea232a4dcfb301d076660f3d5e&role.id=9d68999390534c65b1cfc53d421f6412 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:26.321538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47220f3a-fe3a-41f5-9800-a4b8f2e1b63b None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:26.322078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47220f3a-fe3a-41f5-9800-a4b8f2e1b63b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:26.322078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47220f3a-fe3a-41f5-9800-a4b8f2e1b63b None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:26.323601 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 562/1118] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:26.328547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a5b488-517d-405f-812f-97e4d416a2a2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:26.328950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a5b488-517d-405f-812f-97e4d416a2a2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:26.329080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a5b488-517d-405f-812f-97e4d416a2a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:26.346887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c6a5b488-517d-405f-812f-97e4d416a2a2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:26.348084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c6a5b488-517d-405f-812f-97e4d416a2a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:26.411338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 557/1119] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:26 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:26.680757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47220f3a-fe3a-41f5-9800-a4b8f2e1b63b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:26.750823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:26.782415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47220f3a-fe3a-41f5-9800-a4b8f2e1b63b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:26.815955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:26.816579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:26.816853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:26.817122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] PATH_INFO: `/v3/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:26.818498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] RBAC: Authorizing `identity:get_role(role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:26.819842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:26.821199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 563/1120] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:26 2022] GET /identity/v3/roles/9d68999390534c65b1cfc53d421f6412 => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:26.827221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a5b488-517d-405f-812f-97e4d416a2a2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:26.924691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a5b488-517d-405f-812f-97e4d416a2a2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:26.937467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:26.958639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:26.959251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:26.959525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:26.959789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] PATH_INFO: `/v3/users/e6c64ffa43e54400b9b43123b91cf6e5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:26.960922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] RBAC: Authorizing `identity:get_user(user_id=e6c64ffa43e54400b9b43123b91cf6e5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:26.962340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:26.964458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 558/1121] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:26 2022] GET /identity/v3/users/e6c64ffa43e54400b9b43123b91cf6e5 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:26.970361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:27.066455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a35d857-c085-482b-b670-93c137adb758 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:27.099809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef1e261-70de-4426-826c-40aa0042fe45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:27.100439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef1e261-70de-4426-826c-40aa0042fe45 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:27.100722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef1e261-70de-4426-826c-40aa0042fe45 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:27.100994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef1e261-70de-4426-826c-40aa0042fe45 None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:27.102092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef1e261-70de-4426-826c-40aa0042fe45 None admin] RBAC: Authorizing `identity:get_project(project_id=6dc938ea232a4dcfb301d076660f3d5e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:27.103503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef1e261-70de-4426-826c-40aa0042fe45 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:27.105276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 564/1122] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:26 2022] GET /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:27.111218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:27.208324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403b55d7-15c5-4b49-9567-cd880992e4de None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:27.241074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08fd831-1c01-47b2-9829-f953af172f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:27.241702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08fd831-1c01-47b2-9829-f953af172f61 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:27.241973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08fd831-1c01-47b2-9829-f953af172f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:27.242235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08fd831-1c01-47b2-9829-f953af172f61 None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e/users/e6c64ffa43e54400b9b43123b91cf6e5/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:27.244340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08fd831-1c01-47b2-9829-f953af172f61 None admin] RBAC: Authorizing `identity:create_grant(project_id=6dc938ea232a4dcfb301d076660f3d5e, user_id=e6c64ffa43e54400b9b43123b91cf6e5, role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:27.246531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08fd831-1c01-47b2-9829-f953af172f61 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:27.258466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 559/1123] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:42:27 2022] PUT /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e/users/e6c64ffa43e54400b9b43123b91cf6e5/roles/9d68999390534c65b1cfc53d421f6412 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:42:29.380549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2a12-fc29-4df7-bc5f-b521068a25db None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:29.381081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2a12-fc29-4df7-bc5f-b521068a25db None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:29.381081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2a12-fc29-4df7-bc5f-b521068a25db None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:29.382414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 565/1124] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:29.388568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc8294-ff3b-4889-9306-0fb35110f66d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:29.388988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc8294-ff3b-4889-9306-0fb35110f66d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:29.389081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bc8294-ff3b-4889-9306-0fb35110f66d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:29.408242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-27bc8294-ff3b-4889-9306-0fb35110f66d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:29.409396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-27bc8294-ff3b-4889-9306-0fb35110f66d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:29.442983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:29.475674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 560/1125] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:29.746044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fb2a12-fc29-4df7-bc5f-b521068a25db None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:29.754793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:29.846937 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fb2a12-fc29-4df7-bc5f-b521068a25db None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:29.880785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:29.881402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:29.881683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:29.881952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] PATH_INFO: `/v3/roles/9d68999390534c65b1cfc53d421f6412` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:29.883330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] RBAC: Authorizing `identity:get_role(role_id=9d68999390534c65b1cfc53d421f6412)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:29.884702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:29.886810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 566/1126] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:29 2022] GET /identity/v3/roles/9d68999390534c65b1cfc53d421f6412 => generated 223 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:29.892672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc8294-ff3b-4889-9306-0fb35110f66d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:29.987087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bc8294-ff3b-4889-9306-0fb35110f66d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:30.019830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:30.020401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:30.020675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:30.020939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] PATH_INFO: `/v3/users/e6c64ffa43e54400b9b43123b91cf6e5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:30.022029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] RBAC: Authorizing `identity:get_user(user_id=e6c64ffa43e54400b9b43123b91cf6e5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:30.023427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:30.024965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 561/1127] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:29 2022] GET /identity/v3/users/e6c64ffa43e54400b9b43123b91cf6e5 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:30.030803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:30.103427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:30.127765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8d39a-8ec8-4590-a81d-bbef3068dc2a None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:30.161086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:30.161667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:30.161969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:30.162235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:30.227072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:30.228521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:30.298110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:30.323544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 567/1128] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:30 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:30.329401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:30.367855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:30.425842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8bc341-a87c-43dc-bac7-bc75f84105c1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:30.459137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3800bca6-7104-4270-bb95-76d497abea9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:30.459746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3800bca6-7104-4270-bb95-76d497abea9c None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:30.460054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3800bca6-7104-4270-bb95-76d497abea9c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:30.460324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3800bca6-7104-4270-bb95-76d497abea9c None admin] PATH_INFO: `/v3/projects/6dc938ea232a4dcfb301d076660f3d5e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:30.461415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3800bca6-7104-4270-bb95-76d497abea9c None admin] RBAC: Authorizing `identity:get_project(project_id=6dc938ea232a4dcfb301d076660f3d5e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:30.462830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3800bca6-7104-4270-bb95-76d497abea9c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:30.464535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 562/1129] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:30 2022] GET /identity/v3/projects/6dc938ea232a4dcfb301d076660f3d5e => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:30.470599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:30.566310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ebf0dd-7225-4366-9ab4-bfbb844a20b2 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:30.599341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e97e3-3115-4955-8ef6-c7930b695756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:30.599917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e97e3-3115-4955-8ef6-c7930b695756 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:30.600220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e97e3-3115-4955-8ef6-c7930b695756 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:30.600485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e97e3-3115-4955-8ef6-c7930b695756 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:30.601184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e97e3-3115-4955-8ef6-c7930b695756 None admin] RBAC: Adding query filter params (role.id=9d68999390534c65b1cfc53d421f6412, scope.project.id=6dc938ea232a4dcfb301d076660f3d5e, user.id=e6c64ffa43e54400b9b43123b91cf6e5) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:30.601483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e97e3-3115-4955-8ef6-c7930b695756 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:30.602875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e97e3-3115-4955-8ef6-c7930b695756 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:30.608816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 568/1130] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:30 2022] GET /identity/v3/role_assignments?user.id=e6c64ffa43e54400b9b43123b91cf6e5&scope.project.id=6dc938ea232a4dcfb301d076660f3d5e&role.id=9d68999390534c65b1cfc53d421f6412 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:32.764169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110e7fc6-2974-4c4b-a2ca-1a6a26eb0f57 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:32.764702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110e7fc6-2974-4c4b-a2ca-1a6a26eb0f57 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:32.764702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110e7fc6-2974-4c4b-a2ca-1a6a26eb0f57 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:32.766011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 563/1131] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:32.771194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465f4f0-e97f-4e09-959f-b93a8f542342 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:32.771585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465f4f0-e97f-4e09-959f-b93a8f542342 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:32.771740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465f4f0-e97f-4e09-959f-b93a8f542342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:32.790069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1465f4f0-e97f-4e09-959f-b93a8f542342 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:32.791272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1465f4f0-e97f-4e09-959f-b93a8f542342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:32.820786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:32.858044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 569/1132] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:33.133134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110e7fc6-2974-4c4b-a2ca-1a6a26eb0f57 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:33.231823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110e7fc6-2974-4c4b-a2ca-1a6a26eb0f57 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:33.238883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:33.265633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f823e529-5d6a-4eac-9fac-cc3aa47d2c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:33.266243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823e529-5d6a-4eac-9fac-cc3aa47d2c99 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:33.266516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823e529-5d6a-4eac-9fac-cc3aa47d2c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:33.266779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f823e529-5d6a-4eac-9fac-cc3aa47d2c99 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:33.267856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f823e529-5d6a-4eac-9fac-cc3aa47d2c99 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:33.269288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f823e529-5d6a-4eac-9fac-cc3aa47d2c99 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:33.270804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 564/1133] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:33.277094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465f4f0-e97f-4e09-959f-b93a8f542342 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:33.372037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465f4f0-e97f-4e09-959f-b93a8f542342 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:33.406096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723f48b6-bc40-4cdd-a2ee-2652037c9942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:33.406626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f48b6-bc40-4cdd-a2ee-2652037c9942 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:33.406899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f48b6-bc40-4cdd-a2ee-2652037c9942 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:33.407164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f48b6-bc40-4cdd-a2ee-2652037c9942 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:33.407811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723f48b6-bc40-4cdd-a2ee-2652037c9942 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:33.409241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723f48b6-bc40-4cdd-a2ee-2652037c9942 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:33.435185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 570/1134] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:42:33 2022] POST /identity/v3/projects => generated 311 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:35.575629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a9c2e6-c16e-422b-b6a8-3a91f422e7a6 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:35.576258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a9c2e6-c16e-422b-b6a8-3a91f422e7a6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:35.576258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a9c2e6-c16e-422b-b6a8-3a91f422e7a6 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:35.577538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 565/1135] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:35.582728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4a64b-cb63-4a78-a500-1b7575a262dc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:35.583097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4a64b-cb63-4a78-a500-1b7575a262dc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:35.583266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4a64b-cb63-4a78-a500-1b7575a262dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:35.602284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7cb4a64b-cb63-4a78-a500-1b7575a262dc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:35.603447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7cb4a64b-cb63-4a78-a500-1b7575a262dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:35.617488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:35.669500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 571/1136] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:35.939200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a9c2e6-c16e-422b-b6a8-3a91f422e7a6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:36.038734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a9c2e6-c16e-422b-b6a8-3a91f422e7a6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:36.073807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96005d0-8866-49bd-9019-ad817c772271 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:36.074421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96005d0-8866-49bd-9019-ad817c772271 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:36.074698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96005d0-8866-49bd-9019-ad817c772271 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:36.074965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96005d0-8866-49bd-9019-ad817c772271 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:36.076102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96005d0-8866-49bd-9019-ad817c772271 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:36.077521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96005d0-8866-49bd-9019-ad817c772271 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:36.079052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 566/1137] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:42:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:36.084879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb4a64b-cb63-4a78-a500-1b7575a262dc None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:36.178457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb4a64b-cb63-4a78-a500-1b7575a262dc None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:36.211287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd6a94-a7ca-4537-b0d5-8072ae0e7d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:36.211815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd6a94-a7ca-4537-b0d5-8072ae0e7d52 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:36.212116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd6a94-a7ca-4537-b0d5-8072ae0e7d52 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:36.212385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd6a94-a7ca-4537-b0d5-8072ae0e7d52 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:36.213621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddd6a94-a7ca-4537-b0d5-8072ae0e7d52 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:36.215440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddd6a94-a7ca-4537-b0d5-8072ae0e7d52 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:36.236542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 572/1138] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:42:36 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:38.436629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc012e18-88a2-4864-9c78-5a114e4e2464 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:38.437210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc012e18-88a2-4864-9c78-5a114e4e2464 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:38.437210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc012e18-88a2-4864-9c78-5a114e4e2464 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:38.438498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 567/1139] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:38.443581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8a5c9-6748-4b42-bc86-e0b3a464ab7d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:38.444042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8a5c9-6748-4b42-bc86-e0b3a464ab7d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:38.444145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8a5c9-6748-4b42-bc86-e0b3a464ab7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:38.457289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:38.463702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d5d8a5c9-6748-4b42-bc86-e0b3a464ab7d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:38.465012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d5d8a5c9-6748-4b42-bc86-e0b3a464ab7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:38.528761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 573/1140] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:38.797593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc012e18-88a2-4864-9c78-5a114e4e2464 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:38.842005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:38.898293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc012e18-88a2-4864-9c78-5a114e4e2464 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:38.931056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:38.931789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:38.932129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:38.932392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:38.942150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:38.946919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:38.948954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:38.949953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 568/1141] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:42:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:38.954670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d8a5c9-6748-4b42-bc86-e0b3a464ab7d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:39.049684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d8a5c9-6748-4b42-bc86-e0b3a464ab7d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:39.082219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:39.082800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:39.083076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:39.083342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:39.084013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:39.084310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:39.085645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:39.087725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:39.094785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 574/1142] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:42:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:39.100692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:39.196051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176524e2-e21a-4e3c-9db5-25c93244b895 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:39.198014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:39.230215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:39.230817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:39.231091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:39.231358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] PATH_INFO: `/v3/users/663e65ac104e47d6a5c60e432c6706a8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:39.243994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] RBAC: Authorizing `identity:get_user(user_id=663e65ac104e47d6a5c60e432c6706a8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:39.246071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:39.247612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 569/1143] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:39 2022] GET /identity/v3/users/663e65ac104e47d6a5c60e432c6706a8 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:39.253381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:39.349201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a10427-1653-4e2f-a9d2-858ac0203a2d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:39.382264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:39.382857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:39.383134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:39.383402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:39.444982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:39.449066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:39.450495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:39.545386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 575/1144] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:39.551343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:39.648293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe4f818-fb06-4b8c-9273-de5bb3cc9929 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:39.680890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb6ba70-64ae-42b6-99a4-cacadfddad45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:39.681487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6ba70-64ae-42b6-99a4-cacadfddad45 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:39.682068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6ba70-64ae-42b6-99a4-cacadfddad45 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:39.682379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6ba70-64ae-42b6-99a4-cacadfddad45 None admin] PATH_INFO: `/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:39.683476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb6ba70-64ae-42b6-99a4-cacadfddad45 None admin] RBAC: Authorizing `identity:get_project(project_id=3c79b24a41ff412a9cb4a03dee3ecbd0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:39.684907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb6ba70-64ae-42b6-99a4-cacadfddad45 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:39.686467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 570/1145] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:39 2022] GET /identity/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:39.692536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:39.790470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:39.795491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5dc4c3-8c02-4e1d-89c1-efd629e7ac06 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:39.829094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344fa10c-f050-404c-824e-12ce8dba7cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:39.829682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344fa10c-f050-404c-824e-12ce8dba7cf6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:39.829954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344fa10c-f050-404c-824e-12ce8dba7cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:39.830218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344fa10c-f050-404c-824e-12ce8dba7cf6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:39.830917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344fa10c-f050-404c-824e-12ce8dba7cf6 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=3c79b24a41ff412a9cb4a03dee3ecbd0, user.id=663e65ac104e47d6a5c60e432c6706a8) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:39.831215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344fa10c-f050-404c-824e-12ce8dba7cf6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:39.832640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344fa10c-f050-404c-824e-12ce8dba7cf6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:39.837725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 576/1146] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:39 2022] GET /identity/v3/role_assignments?user.id=663e65ac104e47d6a5c60e432c6706a8&scope.project.id=3c79b24a41ff412a9cb4a03dee3ecbd0&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:41.943822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e166465-f71d-4bb0-a779-7276d6c5c8b4 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:41.944377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e166465-f71d-4bb0-a779-7276d6c5c8b4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:41.944377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e166465-f71d-4bb0-a779-7276d6c5c8b4 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:41.945687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 571/1147] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:41.950782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8337715-e8ee-439c-9888-72ac74898469 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:41.951182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8337715-e8ee-439c-9888-72ac74898469 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:41.951318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8337715-e8ee-439c-9888-72ac74898469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:41.969187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a8337715-e8ee-439c-9888-72ac74898469 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:41.970355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a8337715-e8ee-439c-9888-72ac74898469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:42.034221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 577/1148] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:42.303778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e166465-f71d-4bb0-a779-7276d6c5c8b4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:42.330749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:42.402755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e166465-f71d-4bb0-a779-7276d6c5c8b4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:42.436051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:42.436658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:42.436931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:42.437194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:42.446756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:42.451871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:42.453004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:42.454030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:42.456087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 572/1149] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:42:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:42.458859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8337715-e8ee-439c-9888-72ac74898469 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:42.547837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:42.556824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8337715-e8ee-439c-9888-72ac74898469 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:42.589764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:42.590348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:42.590621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:42.590886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:42.591529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:42.591823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:42.593184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:42.600833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 578/1150] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:42:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:42.607367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:42.693883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:42.706160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd87690b-0ae5-41f3-8d32-5880c982c9ba None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:42.738501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:42.739133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:42.739407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:42.739673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] PATH_INFO: `/v3/users/663e65ac104e47d6a5c60e432c6706a8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:42.740799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] RBAC: Authorizing `identity:get_user(user_id=663e65ac104e47d6a5c60e432c6706a8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:42.742205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:42.743738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 573/1151] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:42 2022] GET /identity/v3/users/663e65ac104e47d6a5c60e432c6706a8 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:42.749689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:42.847911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6b6783-f1a4-4e6f-b891-a9b19613a393 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:42.881140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d2c9d8-e838-400f-8622-288544b4dbd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:42.881759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d2c9d8-e838-400f-8622-288544b4dbd5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:42.882037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d2c9d8-e838-400f-8622-288544b4dbd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:42.882305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d2c9d8-e838-400f-8622-288544b4dbd5 None admin] PATH_INFO: `/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:42.883401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d2c9d8-e838-400f-8622-288544b4dbd5 None admin] RBAC: Authorizing `identity:get_project(project_id=3c79b24a41ff412a9cb4a03dee3ecbd0)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:42.884897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d2c9d8-e838-400f-8622-288544b4dbd5 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:42.886750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 579/1152] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:42 2022] GET /identity/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:42.892744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:42.988732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9c3176-0a67-4558-81d7-cfead4f21a45 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:43.021465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ef1b53-1606-4b76-9c65-32f1657d11fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:43.022096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef1b53-1606-4b76-9c65-32f1657d11fa None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:43.022369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef1b53-1606-4b76-9c65-32f1657d11fa None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:43.022635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ef1b53-1606-4b76-9c65-32f1657d11fa None admin] PATH_INFO: `/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0/users/663e65ac104e47d6a5c60e432c6706a8/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:43.024524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef1b53-1606-4b76-9c65-32f1657d11fa None admin] RBAC: Authorizing `identity:create_grant(project_id=3c79b24a41ff412a9cb4a03dee3ecbd0, user_id=663e65ac104e47d6a5c60e432c6706a8, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:43.026445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ef1b53-1606-4b76-9c65-32f1657d11fa None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:43.039248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 574/1153] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:42:42 2022] PUT /identity/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0/users/663e65ac104e47d6a5c60e432c6706a8/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:42:45.159323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b38aa-52ad-46d0-8fb1-80b13422bee6 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:45.159867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b38aa-52ad-46d0-8fb1-80b13422bee6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:45.159867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9b38aa-52ad-46d0-8fb1-80b13422bee6 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:45.161212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 580/1154] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:45.167089 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4899ac-4fc6-4a77-b76e-77ec26457a42 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:45.167486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4899ac-4fc6-4a77-b76e-77ec26457a42 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:45.167623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4899ac-4fc6-4a77-b76e-77ec26457a42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:45.186135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2e4899ac-4fc6-4a77-b76e-77ec26457a42 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:45.187312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2e4899ac-4fc6-4a77-b76e-77ec26457a42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:45.198990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:45.254071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 575/1155] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:42:45.524241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b38aa-52ad-46d0-8fb1-80b13422bee6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:45.548369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:45.622345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9b38aa-52ad-46d0-8fb1-80b13422bee6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:45.654926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:45.655534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:45.655813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:45.656107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:45.665630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:42:45.670926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:45.672025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:42:45.673004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 581/1156] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:42:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:45.678015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4899ac-4fc6-4a77-b76e-77ec26457a42 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:45.773594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4899ac-4fc6-4a77-b76e-77ec26457a42 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:45.810286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:45.810879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:45.811153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:45.811418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:45.812094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:45.812389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:45.813734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:45.821637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 576/1157] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:42:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:45.827743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:45.907806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:45.926223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab7c325-5cea-4fca-afc9-ec02c8cf891c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:45.959183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:45.959797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:45.960110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:45.960384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] PATH_INFO: `/v3/users/663e65ac104e47d6a5c60e432c6706a8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:45.961486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] RBAC: Authorizing `identity:get_user(user_id=663e65ac104e47d6a5c60e432c6706a8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:45.963040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:45.964631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 582/1158] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:42:45 2022] GET /identity/v3/users/663e65ac104e47d6a5c60e432c6706a8 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:45.970401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:45.982213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:46.069530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47e82b3-cee2-49d1-9c38-ae885b0bd242 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:46.102232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:46.102825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:46.103103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:46.104037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:46.170584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:46.172049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:46.181288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:46.268865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 577/1159] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:46.274804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:46.372912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1805bb0-8dcb-4535-b4ef-6cee472ee8b3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:46.406198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e1481-19ae-49e5-bc2a-cced43585cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:46.406813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e1481-19ae-49e5-bc2a-cced43585cb7 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:46.407089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e1481-19ae-49e5-bc2a-cced43585cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:46.407358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e1481-19ae-49e5-bc2a-cced43585cb7 None admin] PATH_INFO: `/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:46.408475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e1481-19ae-49e5-bc2a-cced43585cb7 None admin] RBAC: Authorizing `identity:get_project(project_id=3c79b24a41ff412a9cb4a03dee3ecbd0)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:46.409873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e1481-19ae-49e5-bc2a-cced43585cb7 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:46.411482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 583/1160] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:42:46 2022] GET /identity/v3/projects/3c79b24a41ff412a9cb4a03dee3ecbd0 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:46.418001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:46.513653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a42520c-068e-4527-bd04-21564303cf5f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:46.522443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:46.547484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087bcf1b-ce3f-4f97-a8a1-bdbe4bbde2e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:46.548106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087bcf1b-ce3f-4f97-a8a1-bdbe4bbde2e5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:46.548385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087bcf1b-ce3f-4f97-a8a1-bdbe4bbde2e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:46.548651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087bcf1b-ce3f-4f97-a8a1-bdbe4bbde2e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:46.549363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087bcf1b-ce3f-4f97-a8a1-bdbe4bbde2e5 None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=3c79b24a41ff412a9cb4a03dee3ecbd0, user.id=663e65ac104e47d6a5c60e432c6706a8) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:46.549666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087bcf1b-ce3f-4f97-a8a1-bdbe4bbde2e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:46.551088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087bcf1b-ce3f-4f97-a8a1-bdbe4bbde2e5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:46.557240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 578/1161] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:42:46 2022] GET /identity/v3/role_assignments?user.id=663e65ac104e47d6a5c60e432c6706a8&scope.project.id=3c79b24a41ff412a9cb4a03dee3ecbd0&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:48.699236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd9efe-d22a-497b-ae06-c0382d67a4f0 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:48.699767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd9efe-d22a-497b-ae06-c0382d67a4f0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:48.699767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdd9efe-d22a-497b-ae06-c0382d67a4f0 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:48.701154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 584/1162] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:48.706216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe8486-be5b-40f0-9d19-a7112bd07a19 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:48.706646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe8486-be5b-40f0-9d19-a7112bd07a19 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:48.706744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbe8486-be5b-40f0-9d19-a7112bd07a19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:48.724353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7bbe8486-be5b-40f0-9d19-a7112bd07a19 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:48.725526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7bbe8486-be5b-40f0-9d19-a7112bd07a19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:48.790470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 579/1163] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:49.058240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdd9efe-d22a-497b-ae06-c0382d67a4f0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:49.076878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:49.157193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdd9efe-d22a-497b-ae06-c0382d67a4f0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:49.189575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197dd833-af25-4b47-b268-216137e7236e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:49.190161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197dd833-af25-4b47-b268-216137e7236e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:49.190436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197dd833-af25-4b47-b268-216137e7236e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:49.190703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197dd833-af25-4b47-b268-216137e7236e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:49.254768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197dd833-af25-4b47-b268-216137e7236e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:49.256218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197dd833-af25-4b47-b268-216137e7236e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:49.273375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:49.350539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 585/1164] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:42:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:42:49.356879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbe8486-be5b-40f0-9d19-a7112bd07a19 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:49.453403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbe8486-be5b-40f0-9d19-a7112bd07a19 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:49.454868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:49.486752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a55170-acee-44ed-b38e-1a4de729962a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:49.487358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a55170-acee-44ed-b38e-1a4de729962a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:49.487634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a55170-acee-44ed-b38e-1a4de729962a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:49.487900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a55170-acee-44ed-b38e-1a4de729962a None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:49.494403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a55170-acee-44ed-b38e-1a4de729962a None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:49.495824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a55170-acee-44ed-b38e-1a4de729962a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-c5a55170-acee-44ed-b38e-1a4de729962a None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:42:49.506935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:42:49.508151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:42:49.509049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 580/1165] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:42:49 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:42:49.514945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197dd833-af25-4b47-b268-216137e7236e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:49.610155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197dd833-af25-4b47-b268-216137e7236e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:49.627052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:49.643784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936d7d18-a7ed-4103-a8a7-49398c9be3fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:49.644409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d7d18-a7ed-4103-a8a7-49398c9be3fc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:49.644686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d7d18-a7ed-4103-a8a7-49398c9be3fc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:49.644953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936d7d18-a7ed-4103-a8a7-49398c9be3fc None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:49.645606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936d7d18-a7ed-4103-a8a7-49398c9be3fc None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:42:49.645904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936d7d18-a7ed-4103-a8a7-49398c9be3fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:49.647242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936d7d18-a7ed-4103-a8a7-49398c9be3fc None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:49.654342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 586/1166] 10.222.0.42 () {56 vars in 1115 bytes} [Fri Sep 16 01:42:49 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:51.778357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8919f61-d6b7-45a6-9fd0-173847837d14 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:51.778911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8919f61-d6b7-45a6-9fd0-173847837d14 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:51.778911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8919f61-d6b7-45a6-9fd0-173847837d14 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:51.780256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 581/1167] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:51.786396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47555ec-0e34-4e41-b087-b35cc19388f7 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:51.786396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47555ec-0e34-4e41-b087-b35cc19388f7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:51.786396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47555ec-0e34-4e41-b087-b35cc19388f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:51.809168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a47555ec-0e34-4e41-b087-b35cc19388f7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:51.810352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a47555ec-0e34-4e41-b087-b35cc19388f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:51.878445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 587/1168] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:52.147444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8919f61-d6b7-45a6-9fd0-173847837d14 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:52.246801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8919f61-d6b7-45a6-9fd0-173847837d14 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:52.280483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972135c0-b79c-4292-898d-16df08c97f93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:52.281066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972135c0-b79c-4292-898d-16df08c97f93 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:52.281344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972135c0-b79c-4292-898d-16df08c97f93 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:52.281611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972135c0-b79c-4292-898d-16df08c97f93 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:52.282263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972135c0-b79c-4292-898d-16df08c97f93 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:52.283618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972135c0-b79c-4292-898d-16df08c97f93 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:52.300203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 582/1169] 10.222.0.42 () {60 vars in 1138 bytes} [Fri Sep 16 01:42:52 2022] POST /identity/v3/domains => generated 267 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:42:54.464876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d3f034-3ee4-4589-a7f5-0411e72eb261 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:54.465406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d3f034-3ee4-4589-a7f5-0411e72eb261 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:54.465406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d3f034-3ee4-4589-a7f5-0411e72eb261 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:54.466776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 588/1170] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:54.471897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568ac94-9ad0-4071-aeb6-616a34bbfbe2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:54.472304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568ac94-9ad0-4071-aeb6-616a34bbfbe2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:54.472474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b568ac94-9ad0-4071-aeb6-616a34bbfbe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:54.490763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b568ac94-9ad0-4071-aeb6-616a34bbfbe2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:54.491951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b568ac94-9ad0-4071-aeb6-616a34bbfbe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:54.507786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:54.557803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 583/1171] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:54.824677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d3f034-3ee4-4589-a7f5-0411e72eb261 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:54.925721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d3f034-3ee4-4589-a7f5-0411e72eb261 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:54.938442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:54.959481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eace52fe-fbb7-4e14-a0e1-0df5a4b73594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:54.960123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eace52fe-fbb7-4e14-a0e1-0df5a4b73594 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:54.960406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eace52fe-fbb7-4e14-a0e1-0df5a4b73594 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:54.960678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eace52fe-fbb7-4e14-a0e1-0df5a4b73594 None admin] PATH_INFO: `/v3/domains/81dca743fa4f40d295c1cfaef6b71869` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:54.970690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eace52fe-fbb7-4e14-a0e1-0df5a4b73594 None admin] RBAC: Authorizing `identity:get_domain(domain_id=81dca743fa4f40d295c1cfaef6b71869)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:54.972149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eace52fe-fbb7-4e14-a0e1-0df5a4b73594 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:54.973712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 589/1172] 10.222.0.42 () {56 vars in 1183 bytes} [Fri Sep 16 01:42:54 2022] GET /identity/v3/domains/81dca743fa4f40d295c1cfaef6b71869 => generated 267 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:54.980122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b568ac94-9ad0-4071-aeb6-616a34bbfbe2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:55.075689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b568ac94-9ad0-4071-aeb6-616a34bbfbe2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:55.109061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6ae180-724d-4940-abc5-4d1e42514bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:55.109643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ae180-724d-4940-abc5-4d1e42514bca None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:55.109922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ae180-724d-4940-abc5-4d1e42514bca None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:55.110190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6ae180-724d-4940-abc5-4d1e42514bca None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:55.110834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6ae180-724d-4940-abc5-4d1e42514bca None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:55.112291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6ae180-724d-4940-abc5-4d1e42514bca None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:55.138431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 584/1173] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:42:54 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:42:57.289933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d89248f-3d6a-4656-93de-c634db629fb4 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:57.291059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d89248f-3d6a-4656-93de-c634db629fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:57.291148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d89248f-3d6a-4656-93de-c634db629fb4 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:57.292508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 590/1174] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:42:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:42:57.297814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c2ffd-3b8f-4208-8f27-9de700a8deaa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:57.298246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c2ffd-3b8f-4208-8f27-9de700a8deaa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:57.298369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4c2ffd-3b8f-4208-8f27-9de700a8deaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:57.305471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:57.317728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bc4c2ffd-3b8f-4208-8f27-9de700a8deaa None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:42:57.318900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bc4c2ffd-3b8f-4208-8f27-9de700a8deaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:42:57.383841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 585/1175] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:42:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:42:57.653178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d89248f-3d6a-4656-93de-c634db629fb4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:57.751856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d89248f-3d6a-4656-93de-c634db629fb4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:57.783908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02d91f7-a396-4cb7-909a-7e6e6d904902 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:57.784541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02d91f7-a396-4cb7-909a-7e6e6d904902 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:57.784815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02d91f7-a396-4cb7-909a-7e6e6d904902 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:57.785081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02d91f7-a396-4cb7-909a-7e6e6d904902 None admin] PATH_INFO: `/v3/domains/81dca743fa4f40d295c1cfaef6b71869` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:57.786169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02d91f7-a396-4cb7-909a-7e6e6d904902 None admin] RBAC: Authorizing `identity:get_domain(domain_id=81dca743fa4f40d295c1cfaef6b71869)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:57.787563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02d91f7-a396-4cb7-909a-7e6e6d904902 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:57.789099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 591/1176] 10.222.0.42 () {56 vars in 1183 bytes} [Fri Sep 16 01:42:57 2022] GET /identity/v3/domains/81dca743fa4f40d295c1cfaef6b71869 => generated 267 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:42:57.795366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c2ffd-3b8f-4208-8f27-9de700a8deaa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:42:57.891922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4c2ffd-3b8f-4208-8f27-9de700a8deaa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:42:57.924924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b612c4f-8dd2-4f55-8eec-d63d47971900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:42:57.925454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b612c4f-8dd2-4f55-8eec-d63d47971900 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:42:57.925733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b612c4f-8dd2-4f55-8eec-d63d47971900 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:42:57.926001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b612c4f-8dd2-4f55-8eec-d63d47971900 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:42:57.926643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b612c4f-8dd2-4f55-8eec-d63d47971900 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:42:57.928092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b612c4f-8dd2-4f55-8eec-d63d47971900 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:42:57.936481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:42:57.950514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 586/1177] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:42:57 2022] POST /identity/v3/users => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:43:00.119611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5819d0b-e6ca-479b-ac5a-5922e4bb19e8 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:00.120148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5819d0b-e6ca-479b-ac5a-5922e4bb19e8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:00.120335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5819d0b-e6ca-479b-ac5a-5922e4bb19e8 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:00.121658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 592/1178] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:43:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:43:00.127039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddc3da-6ace-4d4e-8e69-2e6e9083f6c6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:00.127411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddc3da-6ace-4d4e-8e69-2e6e9083f6c6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:00.127584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ddc3da-6ace-4d4e-8e69-2e6e9083f6c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:00.145601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-97ddc3da-6ace-4d4e-8e69-2e6e9083f6c6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:43:00.146778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-97ddc3da-6ace-4d4e-8e69-2e6e9083f6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:43:00.211768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 587/1179] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:43:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:43:00.485052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5819d0b-e6ca-479b-ac5a-5922e4bb19e8 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:00.527626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:00.584880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5819d0b-e6ca-479b-ac5a-5922e4bb19e8 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:00.617309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:00.617924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:00.618708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:00.618979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:00.628805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:43:00.633620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:43:00.635241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:43:00.636506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 593/1180] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:43:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:43:00.641636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddc3da-6ace-4d4e-8e69-2e6e9083f6c6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:00.737580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ddc3da-6ace-4d4e-8e69-2e6e9083f6c6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:00.757316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:00.772068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:00.772662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:00.772942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:00.773212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:00.773865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:43:00.774164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:00.775770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:00.783570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 588/1181] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:43:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:00.789470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:00.885172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:00.888288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c067c2-de8d-44b4-93e4-05f9cf6d47f8 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:00.921529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:00.922152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:00.922432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:00.922700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] PATH_INFO: `/v3/users/7f1a6281c6934973ab7754101baf3d19` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:00.936423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] RBAC: Authorizing `identity:get_user(user_id=7f1a6281c6934973ab7754101baf3d19)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:00.937840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:00.939442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 594/1182] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:43:00 2022] GET /identity/v3/users/7f1a6281c6934973ab7754101baf3d19 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:43:00.945207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:01.042473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09e53d9-a0b4-4b85-9fa3-5ba8fdaec9b5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:01.074586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:01.075167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:01.075446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:01.075712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:01.123829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:01.142326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:01.143757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:01.239053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 589/1183] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:43:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:43:01.245563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:01.340514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bea735-885a-4c16-af50-5665c55b971b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:01.372784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96124e0-465b-4ab7-a963-ceb5069d1400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:01.373386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96124e0-465b-4ab7-a963-ceb5069d1400 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:01.373656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96124e0-465b-4ab7-a963-ceb5069d1400 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:01.373921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96124e0-465b-4ab7-a963-ceb5069d1400 None admin] PATH_INFO: `/v3/projects/0b3b18ee3d014d19975acba9030513ac` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:01.374995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96124e0-465b-4ab7-a963-ceb5069d1400 None admin] RBAC: Authorizing `identity:get_project(project_id=0b3b18ee3d014d19975acba9030513ac)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:01.376422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96124e0-465b-4ab7-a963-ceb5069d1400 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:01.377997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 595/1184] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:43:01 2022] GET /identity/v3/projects/0b3b18ee3d014d19975acba9030513ac => generated 361 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:01.384610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:01.463523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:01.481813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e002e27-d9fb-46b3-8fdb-71117aefd5a5 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:01.515337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af9b269-b830-4bef-acb9-44adaf0bb61d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:01.515925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af9b269-b830-4bef-acb9-44adaf0bb61d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:01.516231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af9b269-b830-4bef-acb9-44adaf0bb61d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:01.516497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af9b269-b830-4bef-acb9-44adaf0bb61d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:01.517205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af9b269-b830-4bef-acb9-44adaf0bb61d None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=0b3b18ee3d014d19975acba9030513ac, user.id=7f1a6281c6934973ab7754101baf3d19) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:43:01.517507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af9b269-b830-4bef-acb9-44adaf0bb61d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:01.518919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af9b269-b830-4bef-acb9-44adaf0bb61d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:01.524101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 590/1185] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:43:01 2022] GET /identity/v3/role_assignments?user.id=7f1a6281c6934973ab7754101baf3d19&scope.project.id=0b3b18ee3d014d19975acba9030513ac&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:03.632615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500d43c-7eac-40dd-a9ae-18b487906f16 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:03.633173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500d43c-7eac-40dd-a9ae-18b487906f16 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:03.633173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500d43c-7eac-40dd-a9ae-18b487906f16 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:03.634967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 596/1186] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:43:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:43:03.640403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7648029-7461-4217-8ca3-4c4f7c0d610b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:03.640772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7648029-7461-4217-8ca3-4c4f7c0d610b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:03.640948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7648029-7461-4217-8ca3-4c4f7c0d610b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:03.659138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c7648029-7461-4217-8ca3-4c4f7c0d610b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:43:03.660347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c7648029-7461-4217-8ca3-4c4f7c0d610b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:43:03.724177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 591/1187] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:43:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:43:03.991739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500d43c-7eac-40dd-a9ae-18b487906f16 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:04.019105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:04.091005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0500d43c-7eac-40dd-a9ae-18b487906f16 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:04.123596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:04.124235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:04.124518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:04.124785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:04.134533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:43:04.139578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:43:04.140702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:43:04.141844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 597/1188] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:43:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:43:04.146715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7648029-7461-4217-8ca3-4c4f7c0d610b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:04.221816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:04.244848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7648029-7461-4217-8ca3-4c4f7c0d610b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:04.277856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:04.278446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:04.278725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:04.278994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:04.279643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:43:04.279967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:04.281323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:04.289080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 592/1189] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:43:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:04.295544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:04.382290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:04.398367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76094f4-535c-4fee-a601-9c86b078e0c4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:04.432416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:04.433050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:04.433317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:04.433584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] PATH_INFO: `/v3/users/7f1a6281c6934973ab7754101baf3d19` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:04.434690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] RBAC: Authorizing `identity:get_user(user_id=7f1a6281c6934973ab7754101baf3d19)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:04.436129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:04.437686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 598/1190] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:43:04 2022] GET /identity/v3/users/7f1a6281c6934973ab7754101baf3d19 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:04.443459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:04.539814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2503dc19-3421-4953-ab42-41b80da8b01d None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:04.572611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c5c301-9e70-43b7-bd74-9fd566f98d1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:04.573222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c5c301-9e70-43b7-bd74-9fd566f98d1f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:04.573497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c5c301-9e70-43b7-bd74-9fd566f98d1f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:04.573764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c5c301-9e70-43b7-bd74-9fd566f98d1f None admin] PATH_INFO: `/v3/projects/0b3b18ee3d014d19975acba9030513ac` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:04.574859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c5c301-9e70-43b7-bd74-9fd566f98d1f None admin] RBAC: Authorizing `identity:get_project(project_id=0b3b18ee3d014d19975acba9030513ac)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:04.576299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c5c301-9e70-43b7-bd74-9fd566f98d1f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:04.577883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 593/1191] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:43:04 2022] GET /identity/v3/projects/0b3b18ee3d014d19975acba9030513ac => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:04.583768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:04.680271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ae5517-0772-4e80-b57a-cc2e497f30e6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:04.713155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289205fe-c15d-49ec-bf14-9108ee05a6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:04.713786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289205fe-c15d-49ec-bf14-9108ee05a6c9 None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:04.714067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289205fe-c15d-49ec-bf14-9108ee05a6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:04.714337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289205fe-c15d-49ec-bf14-9108ee05a6c9 None admin] PATH_INFO: `/v3/projects/0b3b18ee3d014d19975acba9030513ac/users/7f1a6281c6934973ab7754101baf3d19/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:04.716215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289205fe-c15d-49ec-bf14-9108ee05a6c9 None admin] RBAC: Authorizing `identity:create_grant(project_id=0b3b18ee3d014d19975acba9030513ac, user_id=7f1a6281c6934973ab7754101baf3d19, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:04.717622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289205fe-c15d-49ec-bf14-9108ee05a6c9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:04.730392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 599/1192] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:43:04 2022] PUT /identity/v3/projects/0b3b18ee3d014d19975acba9030513ac/users/7f1a6281c6934973ab7754101baf3d19/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:43:06.857844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbfb867-f7b9-4d85-a7e1-4c7caaeb17ca None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:06.858338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbfb867-f7b9-4d85-a7e1-4c7caaeb17ca None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:06.858433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbfb867-f7b9-4d85-a7e1-4c7caaeb17ca None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:06.859706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 594/1193] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:43:06.865070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb1cb4-fe00-4b4e-b537-24b3940abdbd None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:06.865439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb1cb4-fe00-4b4e-b537-24b3940abdbd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:06.865627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eb1cb4-fe00-4b4e-b537-24b3940abdbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:06.884098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-37eb1cb4-fe00-4b4e-b537-24b3940abdbd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:43:06.885286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-37eb1cb4-fe00-4b4e-b537-24b3940abdbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:43:06.894593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:06.951788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 600/1194] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:43:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:43:07.221579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbfb867-f7b9-4d85-a7e1-4c7caaeb17ca None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:07.230995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:07.322693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbfb867-f7b9-4d85-a7e1-4c7caaeb17ca None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:07.354384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:07.354985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:07.355261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:07.355526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:07.365638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:43:07.370880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:43:07.371977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:43:07.373146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 595/1195] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:43:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:43:07.377938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eb1cb4-fe00-4b4e-b537-24b3940abdbd None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:07.475559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37eb1cb4-fe00-4b4e-b537-24b3940abdbd None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:07.509089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:07.509726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:07.510002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:07.510266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:07.511163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:43:07.511458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:07.512815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:07.520511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 601/1196] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:43:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:07.526730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:07.595124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:07.629786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a0fcd1-c9ea-4c52-b2e9-bdcc19f8aa29 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:07.662035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:07.662641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:07.662913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:07.663177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] PATH_INFO: `/v3/users/7f1a6281c6934973ab7754101baf3d19` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:07.664287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] RBAC: Authorizing `identity:get_user(user_id=7f1a6281c6934973ab7754101baf3d19)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:07.666467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:07.668011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 596/1197] 10.222.0.42 () {56 vars in 1177 bytes} [Fri Sep 16 01:43:07 2022] GET /identity/v3/users/7f1a6281c6934973ab7754101baf3d19 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:07.673827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:07.683461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:07.772646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ed4a20-d0ef-4165-a295-734e67946984 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:07.804165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:07.804779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:07.805054 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:07.805857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:07.871053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:07.872518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:07.879839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:07.967348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 602/1198] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:43:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:43:07.973298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:08.069440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960c2039-d6b2-4aaa-a232-fa111152662f None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:08.101566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7eb6da-4b45-49eb-9b19-bcc028dafbc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:08.102231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7eb6da-4b45-49eb-9b19-bcc028dafbc6 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:08.102574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7eb6da-4b45-49eb-9b19-bcc028dafbc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:08.102860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7eb6da-4b45-49eb-9b19-bcc028dafbc6 None admin] PATH_INFO: `/v3/projects/0b3b18ee3d014d19975acba9030513ac` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:08.103968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7eb6da-4b45-49eb-9b19-bcc028dafbc6 None admin] RBAC: Authorizing `identity:get_project(project_id=0b3b18ee3d014d19975acba9030513ac)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:08.105382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7eb6da-4b45-49eb-9b19-bcc028dafbc6 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:08.106957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 597/1199] 10.222.0.42 () {56 vars in 1186 bytes} [Fri Sep 16 01:43:07 2022] GET /identity/v3/projects/0b3b18ee3d014d19975acba9030513ac => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:08.112944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:43:08.208366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c7b5d1-db32-45cd-a816-d9414d15e306 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:43:08.215675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:08.242438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d6ddfc-6403-46ce-bfdb-00d566b7700c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:43:08.243633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d6ddfc-6403-46ce-bfdb-00d566b7700c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:43:08.243907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d6ddfc-6403-46ce-bfdb-00d566b7700c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:43:08.244206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d6ddfc-6403-46ce-bfdb-00d566b7700c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:43:08.244917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d6ddfc-6403-46ce-bfdb-00d566b7700c None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=0b3b18ee3d014d19975acba9030513ac, user.id=7f1a6281c6934973ab7754101baf3d19) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:43:08.245217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d6ddfc-6403-46ce-bfdb-00d566b7700c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:43:08.246622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d6ddfc-6403-46ce-bfdb-00d566b7700c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:43:08.252251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 603/1200] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:43:08 2022] GET /identity/v3/role_assignments?user.id=7f1a6281c6934973ab7754101baf3d19&scope.project.id=0b3b18ee3d014d19975acba9030513ac&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:43:18.097614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:43:18.249743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=15 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:03.800375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a094864-0355-4f73-9070-910de54f4954 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:03.800938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a094864-0355-4f73-9070-910de54f4954 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:03.800938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a094864-0355-4f73-9070-910de54f4954 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:03.802498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 598/1201] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:03.808193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44688e64-7cca-4d2d-acf3-6c3aa3c25750 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:03.808806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44688e64-7cca-4d2d-acf3-6c3aa3c25750 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:03.808806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44688e64-7cca-4d2d-acf3-6c3aa3c25750 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:03.828219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-44688e64-7cca-4d2d-acf3-6c3aa3c25750 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:03.829404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-44688e64-7cca-4d2d-acf3-6c3aa3c25750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:03.911092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 604/1202] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 16 01:44:04.193379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a094864-0355-4f73-9070-910de54f4954 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:04.345072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a094864-0355-4f73-9070-910de54f4954 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:04.384621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:04.385667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:04.385927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:04.386181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:04.398964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:04.400674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:04.417803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:04.418995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:04.420011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:04.420011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:04.420011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:04.420011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 599/1203] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:04.425016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44688e64-7cca-4d2d-acf3-6c3aa3c25750 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:04.557692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44688e64-7cca-4d2d-acf3-6c3aa3c25750 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:04.598289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:04.606717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01a6027-6447-4b80-a194-bffc4156594e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:04.607331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01a6027-6447-4b80-a194-bffc4156594e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:04.607628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01a6027-6447-4b80-a194-bffc4156594e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:04.607912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01a6027-6447-4b80-a194-bffc4156594e None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:04.608559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01a6027-6447-4b80-a194-bffc4156594e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:04.608901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01a6027-6447-4b80-a194-bffc4156594e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:04.610215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01a6027-6447-4b80-a194-bffc4156594e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:04.627619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 605/1204] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:04.635222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:04.703820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:04.767109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549ec808-bc16-400e-bfff-7ab926631f20 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:04.821765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a95bf65-418d-4698-9d47-4e88602c6209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:04.823232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95bf65-418d-4698-9d47-4e88602c6209 None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:04.823547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95bf65-418d-4698-9d47-4e88602c6209 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:04.823920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a95bf65-418d-4698-9d47-4e88602c6209 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:04.824643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95bf65-418d-4698-9d47-4e88602c6209 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:04.826123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a95bf65-418d-4698-9d47-4e88602c6209 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:04.864263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 600/1205] 10.222.0.42 () {60 vars in 1132 bytes} [Fri Sep 16 01:44:04 2022] POST /identity/v3/users => generated 251 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:44:07.364008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac155b0d-44ea-4fc2-8689-7eb634fb8ae7 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:07.364581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac155b0d-44ea-4fc2-8689-7eb634fb8ae7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:07.364581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac155b0d-44ea-4fc2-8689-7eb634fb8ae7 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:07.366612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 606/1206] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:07.373348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb052f1-e965-48f8-b946-489fba1ff729 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:07.374960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb052f1-e965-48f8-b946-489fba1ff729 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:07.374960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb052f1-e965-48f8-b946-489fba1ff729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:07.395991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bdb052f1-e965-48f8-b946-489fba1ff729 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:07.398042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bdb052f1-e965-48f8-b946-489fba1ff729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:07.486421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 601/1207] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:07.805463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac155b0d-44ea-4fc2-8689-7eb634fb8ae7 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:07.928762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac155b0d-44ea-4fc2-8689-7eb634fb8ae7 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:07.973026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:07.973894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:07.974149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:07.974506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:07.987826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:44:07.995384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:07.996430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:44:07.997298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:08.000194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 607/1208] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:44:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:44:08.003298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb052f1-e965-48f8-b946-489fba1ff729 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:08.057606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:08.128734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb052f1-e965-48f8-b946-489fba1ff729 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:08.168280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:08.169104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:08.169249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:08.169413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:08.170165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:08.170523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:08.172151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:08.182182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 602/1209] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:44:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:08.188126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:08.242091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:08.298237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde57a62-6c7f-4475-8280-665fb286a7bc None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:08.339245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:08.339821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:08.340190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:08.340546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:08.352928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:08.354482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:08.370318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:08.371749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:08.372794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:08.372794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:08.372794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:08.376264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 608/1210] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:08.377697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:08.498720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:08.502949 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeabb18-7080-406f-8eac-50ce38b58c40 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:08.542698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:08.544011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:08.544011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:08.544311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:08.545049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:08.545365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:08.547241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:08.562001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 603/1211] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:08.567944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:08.680907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff6b96e-5383-4c32-8d14-c95370e0c1d6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:08.682611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:08.723143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:08.724315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:08.725702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:08.726397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:08.736629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:08.738875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:44:08.754364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:08.755586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:08.757339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:08.763201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 609/1212] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:08 2022] GET /identity/v3/users/placement => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:08.764918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:08.884568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b344e48e-3375-4b43-8f7f-726a05d849e2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:08.921239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:08.921841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:08.922117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:08.922440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:08.923164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:08.923462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:08.924908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:08.949370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 604/1213] 10.222.0.42 () {56 vars in 1143 bytes} [Fri Sep 16 01:44:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:44:08.950529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:09.071799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8de64ab-dcf5-4a1c-bfcf-1576e319ac89 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:09.108044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:09.108746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:09.109099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:09.109443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:09.142876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:09.186108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:09.187700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:09.301755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 610/1214] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:44:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 16 01:44:09.308239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:09.333484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:09.439280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3c5a7e-86db-4835-8520-eed8b30d2a69 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:09.483767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:09.484937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:09.484937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:09.485144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:09.496570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:09.498084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:09.515222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:09.516582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:09.517799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:09.517799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:09.517799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:09.518972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 605/1215] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:09.524750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:09.601590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:09.652720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ef927d-af48-4e6c-ac5b-484cee9f8e4b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:09.693773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:09.694526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:09.694933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:09.695389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:09.696240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:09.696633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:09.698080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:09.712275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 611/1216] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:09.718611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:09.813864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:09.846909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f6415b-ccd4-472b-ab27-4e071492a275 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:09.887892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:09.888566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:09.888844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:09.889111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:09.897605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:09.899345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:44:09.913349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:09.914542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:09.920388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 606/1217] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:44:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:09.922961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:10.039772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d54ce-b1ab-4a83-897d-2380dfd53990 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:10.065085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:10.077693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a50179-b2a3-4a98-a245-40db07fa9385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:10.078404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a50179-b2a3-4a98-a245-40db07fa9385 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:10.078702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a50179-b2a3-4a98-a245-40db07fa9385 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:10.078979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a50179-b2a3-4a98-a245-40db07fa9385 None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:10.079707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a50179-b2a3-4a98-a245-40db07fa9385 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:10.080069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a50179-b2a3-4a98-a245-40db07fa9385 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:10.081644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a50179-b2a3-4a98-a245-40db07fa9385 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:10.095475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 612/1218] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:44:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:10.102527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:10.209608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe25116-8bbc-44bb-b6c5-c78bdd1052d0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:10.238328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:10.245168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8898e17d-e2e8-4cf9-b3e0-e6991ffc3508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:10.245798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8898e17d-e2e8-4cf9-b3e0-e6991ffc3508 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:10.246115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8898e17d-e2e8-4cf9-b3e0-e6991ffc3508 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:10.246354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8898e17d-e2e8-4cf9-b3e0-e6991ffc3508 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:10.247104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8898e17d-e2e8-4cf9-b3e0-e6991ffc3508 None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=b4311561ef9743988bdf64a039c0628f) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:10.247410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8898e17d-e2e8-4cf9-b3e0-e6991ffc3508 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:10.248925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8898e17d-e2e8-4cf9-b3e0-e6991ffc3508 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:10.254840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 607/1219] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:44:10 2022] GET /identity/v3/role_assignments?user.id=b4311561ef9743988bdf64a039c0628f&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:12.565186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf4ceb0-f455-4cf0-b64b-8131c5c63c8e None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:12.565780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf4ceb0-f455-4cf0-b64b-8131c5c63c8e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:12.565919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf4ceb0-f455-4cf0-b64b-8131c5c63c8e None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:12.567219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 613/1220] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:12.573848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499e1c85-7908-4183-b54a-89fffd7b8b4b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:12.574308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499e1c85-7908-4183-b54a-89fffd7b8b4b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:12.574451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499e1c85-7908-4183-b54a-89fffd7b8b4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:12.594184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-499e1c85-7908-4183-b54a-89fffd7b8b4b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:12.595482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-499e1c85-7908-4183-b54a-89fffd7b8b4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:12.670424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 608/1221] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:12.963359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf4ceb0-f455-4cf0-b64b-8131c5c63c8e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:13.072741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf4ceb0-f455-4cf0-b64b-8131c5c63c8e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:13.109920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:13.110583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:13.110868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:13.111139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:13.122390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:44:13.127346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:13.128459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:44:13.129491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:13.134104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 614/1222] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:44:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:13.135633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499e1c85-7908-4183-b54a-89fffd7b8b4b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:13.247385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499e1c85-7908-4183-b54a-89fffd7b8b4b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:13.273937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:13.286180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:13.287399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:13.287679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:13.287963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:13.288648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] RBAC: Adding query filter params (name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:13.288947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:13.290318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:13.298786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 609/1223] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:44:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:13.304924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:13.353626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:13.421282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b4644-1c7e-4623-8bdc-71e49d1ae4c5 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:13.460673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:13.461301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:13.461575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:13.461836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:13.472707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:13.474884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:13.490490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:13.491625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:13.497599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:13.497599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:13.497599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:13.497599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 615/1224] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 16 01:44:13.498352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:13.606563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79274010-6504-47ed-8230-18600bac5ebc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:13.640940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:13.641535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:13.641813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:13.642081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:13.642707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:13.642996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:13.644381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:13.661410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 610/1225] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:13.663185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:13.768127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:13.776378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f243eaa5-16cc-4194-9dd9-5a70acfe2b7f None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:13.813936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:13.814676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:13.814999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:13.815286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:13.824865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:13.826503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:44:13.839635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:13.840837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:13.841813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 616/1226] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:13 2022] GET /identity/v3/users/placement => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:13.846993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:13.899409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:13.959528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9622a6-3c90-46f9-8250-90c6bf41b974 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:13.995677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:13.996309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:13.996682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:13.997031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:13.997822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:13.998196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:13.999765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:14.017898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 611/1227] 10.222.0.42 () {56 vars in 1143 bytes} [Fri Sep 16 01:44:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:14.024464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:14.130355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51807bb3-6f2d-4e79-ba91-3de427469e25 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:14.168618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:14.169299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:14.169596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:14.169850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:14.181870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:14.183302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:14.198277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:14.199737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:14.201062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:14.201062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:14.201062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:14.204173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 617/1228] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:14.205728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:14.316163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3711ad5d-8f03-4334-83d8-2fa6e9a28ec2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:14.319561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:14.357717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:14.358420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:14.358775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:14.359048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:14.359685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:14.360009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:14.361544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:14.375719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 612/1229] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:14.381885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:14.392844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:14.503899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db7ba14-0300-4c6d-94af-cfad0543b075 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:14.540007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:14.540631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:14.540908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:14.541178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:14.547729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:14.549190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:44:14.560912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:14.561812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:14.562913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 618/1230] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:44:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:14.568349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:14.675846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12b293f-95f9-480a-842c-158a6f6cccf2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:14.712326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcde5209-e619-43c8-b572-40f4d2bb09a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:14.712930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcde5209-e619-43c8-b572-40f4d2bb09a8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:14.713275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcde5209-e619-43c8-b572-40f4d2bb09a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:14.713473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcde5209-e619-43c8-b572-40f4d2bb09a8 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:14.714154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcde5209-e619-43c8-b572-40f4d2bb09a8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:14.714497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcde5209-e619-43c8-b572-40f4d2bb09a8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:14.715984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcde5209-e619-43c8-b572-40f4d2bb09a8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:14.729334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 613/1231] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:44:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:14.736550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:14.813225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:14.853186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a37c6e-fb1d-483d-93ca-991e3013848b None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:14.892984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52c7285-a273-46f1-a057-d37e3eacb7ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:14.893647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52c7285-a273-46f1-a057-d37e3eacb7ef None admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:14.894010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52c7285-a273-46f1-a057-d37e3eacb7ef None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:14.894194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52c7285-a273-46f1-a057-d37e3eacb7ef None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/b4311561ef9743988bdf64a039c0628f/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:14.910175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52c7285-a273-46f1-a057-d37e3eacb7ef None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=b4311561ef9743988bdf64a039c0628f, role_id=cb16002d0a2b4241b1d9c4c8ecd9ba42)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:14.911744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52c7285-a273-46f1-a057-d37e3eacb7ef None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:14.924198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 619/1232] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:44:14 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/b4311561ef9743988bdf64a039c0628f/roles/cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:44:17.318875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fca65d3-a22a-4834-b89e-d62a2253d850 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:17.319511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fca65d3-a22a-4834-b89e-d62a2253d850 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:17.319633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fca65d3-a22a-4834-b89e-d62a2253d850 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:17.320982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 614/1233] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:17.327042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc991d2f-59c3-4e44-9879-168e701b7fde None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:17.327461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc991d2f-59c3-4e44-9879-168e701b7fde None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:17.327582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc991d2f-59c3-4e44-9879-168e701b7fde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:17.348919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-dc991d2f-59c3-4e44-9879-168e701b7fde None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:17.350315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-dc991d2f-59c3-4e44-9879-168e701b7fde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:17.435671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 620/1234] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:17.704944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fca65d3-a22a-4834-b89e-d62a2253d850 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:17.730711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:17.814673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fca65d3-a22a-4834-b89e-d62a2253d850 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:17.853418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:17.854101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:17.854406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:17.854697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:17.866551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:44:17.872055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:17.873235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:17.874097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 615/1235] 10.222.0.42 () {56 vars in 1102 bytes} [Fri Sep 16 01:44:17 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:17.879406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc991d2f-59c3-4e44-9879-168e701b7fde None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:17.932437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:17.994461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc991d2f-59c3-4e44-9879-168e701b7fde None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:18.039230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:18.039862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:18.040171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:18.040437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:18.041120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] RBAC: Adding query filter params (name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:18.041497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:18.042854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:18.050816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 621/1236] 10.222.0.42 () {56 vars in 1103 bytes} [Fri Sep 16 01:44:17 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:44:18.057877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:18.159056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:18.176119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24e3f6c-ed82-4456-8f03-11e8d012e226 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:18.214648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:18.215702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:18.216220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:18.216685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:18.231000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:18.232584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:18.248943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:18.254926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:18.257011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:18.257011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:18.257011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:18.257011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 616/1237] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:18.257253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:18.369308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a442a4-5f3c-4438-a45d-1430d4bc6c6c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:18.379015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:18.406911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:18.407745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:18.408080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:18.408430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:18.409112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:18.409413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:18.410963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:18.428554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 622/1238] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:18.434968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:18.548845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6f2002-8725-4fc4-b6c3-b8497bef9b39 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:18.586182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:18.586805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:18.587090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:18.587355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:18.591713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:18.597018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:18.598693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:44:18.610797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:18.611847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:18.612818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 617/1239] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:18 2022] GET /identity/v3/users/placement => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:18.618772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:18.728710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e469569-651b-4720-a87c-abb899f60b61 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:18.767566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:18.768277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:18.768543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:18.768878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:18.769715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:18.770022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:18.772381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:18.792501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 623/1240] 10.222.0.42 () {56 vars in 1143 bytes} [Fri Sep 16 01:44:18 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:18.798388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:18.934067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c8d1af-c9ec-4bf6-a020-353b06a23550 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:18.980066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:18.980660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:18.980939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:18.981205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:19.050470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:19.059040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:19.060579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:19.167505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 618/1241] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:44:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 16 01:44:19.173484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:19.203790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:19.282891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766591f9-f7bd-4d28-aa6f-b54c5accd867 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:19.318924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:19.319536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:19.319817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:19.320117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:19.330814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:19.332280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:19.347457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:19.348534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:19.350262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:19.350262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:19.350262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:19.350262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 624/1242] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:19.355353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:19.467751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:19.475818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875ae13f-0223-43f1-86a6-8e20b0f34c3e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:19.515761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:19.516388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:19.516661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:19.516933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:19.517595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:19.517890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:19.519405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:19.533686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 619/1243] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:19.539915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:19.627334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:19.649580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f79e2b-9663-4307-b9f9-b33bb5890724 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:19.684877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:19.685510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:19.685897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:19.686208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:19.693135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:19.694587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:44:19.705842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:19.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:19.713783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 625/1244] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:44:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:19.714380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:19.826933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16053246-caa8-4c54-a304-fcc848643fe0 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:19.864088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3506f59-8872-41b3-9e6c-2c68ce506169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:19.865478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3506f59-8872-41b3-9e6c-2c68ce506169 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:19.866069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3506f59-8872-41b3-9e6c-2c68ce506169 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:19.866567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3506f59-8872-41b3-9e6c-2c68ce506169 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:19.867266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3506f59-8872-41b3-9e6c-2c68ce506169 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:19.867574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3506f59-8872-41b3-9e6c-2c68ce506169 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:19.869122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3506f59-8872-41b3-9e6c-2c68ce506169 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:19.882178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 620/1245] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:44:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:44:19.888692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:20.000764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479fab47-cbbf-493b-b4c0-17b570053150 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:20.042283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8132c1c-db9e-4e01-b44e-02d489db4a7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:20.042882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8132c1c-db9e-4e01-b44e-02d489db4a7c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:20.043194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8132c1c-db9e-4e01-b44e-02d489db4a7c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:20.043429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8132c1c-db9e-4e01-b44e-02d489db4a7c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:20.044221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8132c1c-db9e-4e01-b44e-02d489db4a7c None admin] RBAC: Adding query filter params (role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=b4311561ef9743988bdf64a039c0628f) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:20.044538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8132c1c-db9e-4e01-b44e-02d489db4a7c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:20.045961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8132c1c-db9e-4e01-b44e-02d489db4a7c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:20.052572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 626/1246] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:44:19 2022] GET /identity/v3/role_assignments?user.id=b4311561ef9743988bdf64a039c0628f&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=cb16002d0a2b4241b1d9c4c8ecd9ba42 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:22.320481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f252c515-db1c-4265-9b7b-7ee2bc375640 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:22.321110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f252c515-db1c-4265-9b7b-7ee2bc375640 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:22.321110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f252c515-db1c-4265-9b7b-7ee2bc375640 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:22.322480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 621/1247] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:22.328360 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88f1f35-aadc-4d24-9add-da09d4bf23ce None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:22.328934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88f1f35-aadc-4d24-9add-da09d4bf23ce None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:22.328934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88f1f35-aadc-4d24-9add-da09d4bf23ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:22.336004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:22.348642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c88f1f35-aadc-4d24-9add-da09d4bf23ce None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:22.350423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c88f1f35-aadc-4d24-9add-da09d4bf23ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:22.424269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 627/1248] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:22.716376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f252c515-db1c-4265-9b7b-7ee2bc375640 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:22.736429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:22.834364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f252c515-db1c-4265-9b7b-7ee2bc375640 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:22.873934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:22.874952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:22.875412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:22.875795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:22.888831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:44:22.894099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:22.895386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:22.897018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 622/1249] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:44:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:22.902796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88f1f35-aadc-4d24-9add-da09d4bf23ce None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:23.008187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88f1f35-aadc-4d24-9add-da09d4bf23ce None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:23.040960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:23.046689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:23.047306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:23.047585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:23.047852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:23.048549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:23.048849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:23.050220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:23.060069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 628/1250] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:44:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:23.066092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:23.146360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:23.171817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4025a3e-c5c3-4a8c-a4cc-5889fcce01b2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:23.207844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:23.208709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:23.208985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:23.209248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:23.220256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:23.221728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:23.238110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:23.239315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:23.240978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:23.240978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:23.240978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:23.244255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 623/1251] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:23.245863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:23.349625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc615b24-bef1-41d2-8b40-e87b901f81ba None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:23.383273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:23.383874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:23.384188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:23.384461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:23.385349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:23.385638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:23.387065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:23.398894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 629/1252] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:23.405500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:23.519971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665bbbca-9021-420a-8792-70c0369d8ab1 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:23.547383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:23.558042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:23.558666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:23.558943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:23.559216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:23.566938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:23.569088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:44:23.579956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:23.581067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:23.582220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 624/1253] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:23 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:23.587251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:23.641346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:23.684970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54c5f3d-2a17-4577-af5e-97b93cc524f9 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:23.718339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:23.718930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:23.719206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:23.719473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:23.720222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:23.720522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:23.721918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:23.738817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 630/1254] 10.222.0.42 () {56 vars in 1143 bytes} [Fri Sep 16 01:44:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:44:23.744692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:23.842751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d764a8aa-74fe-4319-8b71-51e5ca3ce2b4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:23.875023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:23.875609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:23.875887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:23.876182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:23.929825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:23.941945 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:23.944021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:24.039451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 625/1255] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:44:23 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:44:24.045213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:24.142888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bf034b-9761-4273-954a-043d6a740bd6 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:24.155819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:24.176693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:24.177928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:24.178249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:24.178517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:24.188124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:24.189476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:24.204220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:24.205308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:24.206404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:24.206404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:24.206404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:24.206404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 631/1256] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:24.211743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:24.295149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:24.307511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6aacfc-a88a-454b-b4fa-4943ebf0c6ce None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:24.340121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:24.340708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:24.340981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:24.341245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:24.341864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:24.342158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:24.343494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:24.354292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 626/1257] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:24.359869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:24.467497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295a4429-7729-4342-8af4-a19e4d4282d4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:24.500117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:24.500737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:24.501016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:24.501287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:24.508274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:24.509943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:44:24.521770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:24.522819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:24.523709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 632/1258] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:44:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:24.528898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:24.625163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c514d4c7-b355-4800-8e6c-c3918788dec7 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:24.657694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9cb2f8-dd54-4514-885a-d5404b129bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:24.658273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9cb2f8-dd54-4514-885a-d5404b129bf3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:24.658545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9cb2f8-dd54-4514-885a-d5404b129bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:24.658808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9cb2f8-dd54-4514-885a-d5404b129bf3 None admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:24.659474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9cb2f8-dd54-4514-885a-d5404b129bf3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:24.659772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9cb2f8-dd54-4514-885a-d5404b129bf3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:24.661750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9cb2f8-dd54-4514-885a-d5404b129bf3 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:24.673409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 627/1259] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:44:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:24.680169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:24.705869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:24.778073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49848b66-95b4-46a3-b531-86decb49443d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:24.811187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559638b3-6ec4-4eee-a2e5-154fc2a435ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:24.811781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559638b3-6ec4-4eee-a2e5-154fc2a435ae None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:24.812085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559638b3-6ec4-4eee-a2e5-154fc2a435ae None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:24.812349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559638b3-6ec4-4eee-a2e5-154fc2a435ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:24.813056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559638b3-6ec4-4eee-a2e5-154fc2a435ae None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=b4311561ef9743988bdf64a039c0628f) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:24.813354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559638b3-6ec4-4eee-a2e5-154fc2a435ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:24.814753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559638b3-6ec4-4eee-a2e5-154fc2a435ae None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:24.820466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 633/1260] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:44:24 2022] GET /identity/v3/role_assignments?user.id=b4311561ef9743988bdf64a039c0628f&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:27.060584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3821dfbe-78c2-46c4-a303-5bf66711dc98 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:27.061083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3821dfbe-78c2-46c4-a303-5bf66711dc98 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:27.061083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3821dfbe-78c2-46c4-a303-5bf66711dc98 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:27.062354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 628/1261] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:27.068264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5004b73-f063-4cf9-9bcc-f76f5fea84bb None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:27.068690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5004b73-f063-4cf9-9bcc-f76f5fea84bb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:27.068690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5004b73-f063-4cf9-9bcc-f76f5fea84bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:27.088105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f5004b73-f063-4cf9-9bcc-f76f5fea84bb None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:27.089291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f5004b73-f063-4cf9-9bcc-f76f5fea84bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:27.153772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 634/1262] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:27.427922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3821dfbe-78c2-46c4-a303-5bf66711dc98 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:27.439564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:27.536083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3821dfbe-78c2-46c4-a303-5bf66711dc98 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:27.569871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:27.570479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:27.570752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:27.571015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:27.580971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:44:27.586499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:27.587617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:27.588520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 629/1263] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:44:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:27.593653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5004b73-f063-4cf9-9bcc-f76f5fea84bb None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:27.611424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:27.690344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5004b73-f063-4cf9-9bcc-f76f5fea84bb None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:27.723405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:27.724177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:27.724464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:27.724736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:27.725424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:27.725725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:27.727073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:27.734870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 635/1264] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:44:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:27.740698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:27.806869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:27.842091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062ea0bc-7e5d-4c1d-8e03-45514bd6baa3 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:27.873924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:27.874531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:27.874803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:27.875063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:27.885897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:27.887305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:27.901595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:27.902690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:27.903604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:27.903604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:27.903604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:27.903604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 630/1265] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:27.908444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:27.987021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:28.006093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9630dfb5-79d5-4c1b-a537-c7f242ea5072 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:28.039314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:28.039901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:28.040214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:28.040485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:28.041120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:28.041425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:28.042800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:28.054074 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 636/1266] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:28.059700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:28.157772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2595ad5-5c12-452e-8e4d-c46d86932839 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:28.176481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:28.190771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:28.191387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:28.191664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:28.191954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] PATH_INFO: `/v3/users/placement` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:28.199195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:28.200648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:44:28.212044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:28.213152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:28.214434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 631/1267] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:28.218920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:28.314794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d6e0fd-eeb1-4af2-8aaf-9426ff3ef95c None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:28.346966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:28.347546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:28.347821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:28.348120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:28.349440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:28.349736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:28.351164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:28.359376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:28.368482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 637/1268] 10.222.0.42 () {56 vars in 1143 bytes} [Fri Sep 16 01:44:28 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:28.374284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:28.477230 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112613b6-d6d3-4ec8-877c-5751a321d1fc None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:28.509555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:28.510418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:28.510693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:28.510957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:28.521633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:28.523050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:28.536895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:28.537986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:28.539123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:28.539123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:28.539123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:28.539123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 632/1269] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:28.543843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:28.638926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f77eaff-fe9f-4b24-9d12-1446068f7876 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:28.671246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:28.671837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:28.672145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:28.672413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:28.673037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:28.673331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:28.674674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:28.685465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 638/1270] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:28.691077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:28.722628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:28.790123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f19041-decb-4dea-a809-ef3543885d7a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:28.823731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:28.824390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:28.824669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:28.824938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:28.831339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:28.832798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:44:28.844508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:28.845528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:28.846800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 633/1271] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:44:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:28.852052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:28.908581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:28.949934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d0478-4f03-48ec-b9db-533710fd17c3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:28.982961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236eb182-d630-439c-8908-6770b0825dbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:28.983552 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236eb182-d630-439c-8908-6770b0825dbe None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:28.983829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236eb182-d630-439c-8908-6770b0825dbe None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:28.984124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236eb182-d630-439c-8908-6770b0825dbe None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:28.984802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236eb182-d630-439c-8908-6770b0825dbe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:28.985101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236eb182-d630-439c-8908-6770b0825dbe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:28.986501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236eb182-d630-439c-8908-6770b0825dbe None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:28.998039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 639/1272] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:44:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:29.003956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:29.093798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:29.103534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12897ffa-0ec7-4220-a06e-40c19348561a None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:29.136918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e26a237-959f-4c48-b195-82243eb52d06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:29.137555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26a237-959f-4c48-b195-82243eb52d06 None admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:29.137830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26a237-959f-4c48-b195-82243eb52d06 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:29.138097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e26a237-959f-4c48-b195-82243eb52d06 None admin] PATH_INFO: `/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/b4311561ef9743988bdf64a039c0628f/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:29.139972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e26a237-959f-4c48-b195-82243eb52d06 None admin] RBAC: Authorizing `identity:create_grant(project_id=967b318932ae4b66bac5ab6a6be87e75, user_id=b4311561ef9743988bdf64a039c0628f, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:29.141386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e26a237-959f-4c48-b195-82243eb52d06 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:29.153788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 634/1273] 10.222.0.42 () {58 vars in 1439 bytes} [Fri Sep 16 01:44:29 2022] PUT /identity/v3/projects/967b318932ae4b66bac5ab6a6be87e75/users/b4311561ef9743988bdf64a039c0628f/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:44:31.264944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a6b72-3ee7-4e2c-93df-6763c6dcc865 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:31.265504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a6b72-3ee7-4e2c-93df-6763c6dcc865 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:31.265504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0a6b72-3ee7-4e2c-93df-6763c6dcc865 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:31.266821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 640/1274] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:31.272781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7846db-90fa-41d7-9a37-2ffe2c53889f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:31.273202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7846db-90fa-41d7-9a37-2ffe2c53889f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:31.273306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7846db-90fa-41d7-9a37-2ffe2c53889f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:31.293623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8a7846db-90fa-41d7-9a37-2ffe2c53889f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:31.295734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8a7846db-90fa-41d7-9a37-2ffe2c53889f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:31.364791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 635/1275] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:31.642562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0a6b72-3ee7-4e2c-93df-6763c6dcc865 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:31.741623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0a6b72-3ee7-4e2c-93df-6763c6dcc865 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:31.757953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:31.775476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:31.776111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:31.776385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:31.776650 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:31.786802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 16 01:44:31.791586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:31.792692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:31.793878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 641/1276] 10.222.0.42 () {56 vars in 1096 bytes} [Fri Sep 16 01:44:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:31.798675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7846db-90fa-41d7-9a37-2ffe2c53889f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:31.876810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:31.896093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7846db-90fa-41d7-9a37-2ffe2c53889f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:31.930550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:31.931137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:31.931414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:31.931680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:31.932359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] RBAC: Adding query filter params (name=admin) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:31.932658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:31.934009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:31.941713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 636/1277] 10.222.0.42 () {56 vars in 1099 bytes} [Fri Sep 16 01:44:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:31.947816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:32.043644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37477c9b-a543-4362-b14a-1f3497291ae4 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:32.075489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:32.076118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:32.076393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:32.076659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:32.086655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:32.088083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:32.102062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:32.103038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:32.104075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:32.104075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:32.104075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:32.104075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 642/1278] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:32.108673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:32.207896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7113d8c-45ee-498f-88ad-7605048152bf None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:32.240851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:32.241436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:32.241711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:32.241979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:32.242602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:32.242900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:32.244277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:32.255580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 637/1279] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:32.261654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:32.292488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:32.359629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1e961c-0cc2-490e-a1fa-150758fcc293 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:32.392294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:32.392900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:32.393182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:32.393447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:32.400685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:32.402196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 16 01:44:32.414785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:32.415912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:32.416871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 643/1280] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:32 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:32.421758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:32.425477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:32.518988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786b3159-2487-4d98-8fd7-e0ef55afb75e None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:32.552729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:32.553317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:32.553590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:32.553853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:32.554555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:32.554851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:32.556272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:32.572636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 638/1281] 10.222.0.42 () {56 vars in 1143 bytes} [Fri Sep 16 01:44:32 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:32.578410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:32.666274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:32.675751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50848ddb-db39-448b-92c9-7833ac471a0d None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:32.707293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:32.708543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:32.708811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:32.709075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:32.773486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:32.774850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:32.860265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:32.870565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 644/1282] 10.222.0.42 () {58 vars in 1282 bytes} [Fri Sep 16 01:44:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 16 01:44:32.876493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:32.936125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:32.974476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4876f8e-88fe-4139-99fb-61a530eb13b2 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:33.007479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:33.008108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:33.008383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:33.008647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:33.018346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:33.019766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 16 01:44:33.033786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:33.034858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 16 01:44:33.036640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 16 01:44:33.036640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 16 01:44:33.036640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:33.036640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 639/1283] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:44:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:33.041072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:33.137028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfc4280-3c44-4056-96f4-50ef42e77b1e None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:33.169688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:33.170269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:33.170541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:33.170804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:33.171423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:33.171715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:33.173074 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:33.184553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 645/1284] 10.222.0.42 () {56 vars in 1109 bytes} [Fri Sep 16 01:44:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:33.190102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:33.288587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2573b401-7b94-4919-896b-828e324f85c4 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:33.301410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:33.321399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:33.321997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:33.322272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:33.322536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:33.328856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:33.330262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 16 01:44:33.341995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:33.343082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: ERROR keystone.server.flask.application Sep 16 01:44:33.344217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 640/1285] 10.222.0.42 () {56 vars in 1111 bytes} [Fri Sep 16 01:44:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:33.349272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:33.398715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:33.449699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d182b65-9386-48c9-a613-73d4491479cb None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:33.482451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6914096e-9df0-45a0-93b8-5290b61a313a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:33.483042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6914096e-9df0-45a0-93b8-5290b61a313a None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:33.483318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6914096e-9df0-45a0-93b8-5290b61a313a None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:33.483587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6914096e-9df0-45a0-93b8-5290b61a313a None admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:33.484292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6914096e-9df0-45a0-93b8-5290b61a313a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:33.484593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6914096e-9df0-45a0-93b8-5290b61a313a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:33.485988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6914096e-9df0-45a0-93b8-5290b61a313a None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:33.498666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 646/1286] 10.222.0.42 () {56 vars in 1148 bytes} [Fri Sep 16 01:44:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:33.505282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:33.602803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ba399f-f0f6-4816-9077-a1da72d6e989 None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:33.635883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458e26f0-dd0f-4231-864c-edefc1748e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:33.636503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e26f0-dd0f-4231-864c-edefc1748e4d None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:33.636778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e26f0-dd0f-4231-864c-edefc1748e4d None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:33.637043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e26f0-dd0f-4231-864c-edefc1748e4d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:33.637760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458e26f0-dd0f-4231-864c-edefc1748e4d None admin] RBAC: Adding query filter params (role.id=ea4e2ba4a97343b494b1995be6f4e1dc, scope.project.id=967b318932ae4b66bac5ab6a6be87e75, user.id=b4311561ef9743988bdf64a039c0628f) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:33.638062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458e26f0-dd0f-4231-864c-edefc1748e4d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:33.639473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458e26f0-dd0f-4231-864c-edefc1748e4d None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:33.645393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 641/1287] 10.222.0.42 () {56 vars in 1374 bytes} [Fri Sep 16 01:44:33 2022] GET /identity/v3/role_assignments?user.id=b4311561ef9743988bdf64a039c0628f&scope.project.id=967b318932ae4b66bac5ab6a6be87e75&role.id=ea4e2ba4a97343b494b1995be6f4e1dc => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:35.799522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c9209-7d4c-40b6-a718-d629b1a72cad None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:35.800082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c9209-7d4c-40b6-a718-d629b1a72cad None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:35.800082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12c9209-7d4c-40b6-a718-d629b1a72cad None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:35.802200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 647/1288] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:35.807502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79bebf-d5d5-47fd-942f-4e4cbfce1ca6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:35.807969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79bebf-d5d5-47fd-942f-4e4cbfce1ca6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:35.808107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef79bebf-d5d5-47fd-942f-4e4cbfce1ca6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:35.826601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ef79bebf-d5d5-47fd-942f-4e4cbfce1ca6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:35.827754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ef79bebf-d5d5-47fd-942f-4e4cbfce1ca6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:35.856064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:35.893546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 642/1289] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:36.163371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c9209-7d4c-40b6-a718-d629b1a72cad None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:36.265930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12c9209-7d4c-40b6-a718-d629b1a72cad None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:36.276156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:36.298834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:36.299452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:36.299729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:36.300023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] PATH_INFO: `/v3/services/placement` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:36.300598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:36.302471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:36.310914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:36.312136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:36.313183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 648/1290] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:44:36 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:36.318765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef79bebf-d5d5-47fd-942f-4e4cbfce1ca6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:36.416814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef79bebf-d5d5-47fd-942f-4e4cbfce1ca6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:36.460740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a63eaea-daec-44fb-8431-0fa1dd7d0812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:36.461323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a63eaea-daec-44fb-8431-0fa1dd7d0812 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:36.461609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a63eaea-daec-44fb-8431-0fa1dd7d0812 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:36.461877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a63eaea-daec-44fb-8431-0fa1dd7d0812 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:36.462487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a63eaea-daec-44fb-8431-0fa1dd7d0812 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:36.462791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a63eaea-daec-44fb-8431-0fa1dd7d0812 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:36.464218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a63eaea-daec-44fb-8431-0fa1dd7d0812 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:36.469526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 643/1291] 10.222.0.42 () {56 vars in 1116 bytes} [Fri Sep 16 01:44:36 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:44:36.476248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:36.577124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1b678-ef97-4453-8d59-ed71a9806daf None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:36.609051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886fba2e-c55d-4f3d-ae80-70314faab146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:36.609636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886fba2e-c55d-4f3d-ae80-70314faab146 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:36.609910 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886fba2e-c55d-4f3d-ae80-70314faab146 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:36.610174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886fba2e-c55d-4f3d-ae80-70314faab146 None admin] PATH_INFO: `/v3/services` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:36.610782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886fba2e-c55d-4f3d-ae80-70314faab146 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:36.611072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886fba2e-c55d-4f3d-ae80-70314faab146 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:36.612444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886fba2e-c55d-4f3d-ae80-70314faab146 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:36.617402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 649/1292] 10.222.0.42 () {56 vars in 1116 bytes} [Fri Sep 16 01:44:36 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:44:38.751377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff4ae79-eece-4b89-b0d4-bbfe41be8565 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:38.751907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff4ae79-eece-4b89-b0d4-bbfe41be8565 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:38.751907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff4ae79-eece-4b89-b0d4-bbfe41be8565 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:38.753348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 644/1293] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:38.759222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680cd456-3c53-4677-871a-80c3fa43a9cc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:38.759848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680cd456-3c53-4677-871a-80c3fa43a9cc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:38.760048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680cd456-3c53-4677-871a-80c3fa43a9cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:38.777921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-680cd456-3c53-4677-871a-80c3fa43a9cc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:38.779079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-680cd456-3c53-4677-871a-80c3fa43a9cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:38.810916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:38.847907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 650/1294] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:39.115239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff4ae79-eece-4b89-b0d4-bbfe41be8565 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:39.145966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:39.220648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff4ae79-eece-4b89-b0d4-bbfe41be8565 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:39.254059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b2cb4c-5d70-4dd5-af6f-1a9078009e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:39.254689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b2cb4c-5d70-4dd5-af6f-1a9078009e2c None admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:39.254964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b2cb4c-5d70-4dd5-af6f-1a9078009e2c None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:39.255234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b2cb4c-5d70-4dd5-af6f-1a9078009e2c None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:39.255876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b2cb4c-5d70-4dd5-af6f-1a9078009e2c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:39.257255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b2cb4c-5d70-4dd5-af6f-1a9078009e2c None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:39.271907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 645/1295] 10.222.0.42 () {60 vars in 1141 bytes} [Fri Sep 16 01:44:39 2022] POST /identity/v3/services => generated 246 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:44:41.450632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75802259-dfb1-4e0f-96a1-acc1db0cc5c9 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:41.451227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75802259-dfb1-4e0f-96a1-acc1db0cc5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:41.451227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75802259-dfb1-4e0f-96a1-acc1db0cc5c9 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:41.452534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 651/1296] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:41.458707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08babec-babc-4977-903c-89685a906d27 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:41.459467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08babec-babc-4977-903c-89685a906d27 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:41.459467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08babec-babc-4977-903c-89685a906d27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:41.477594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d08babec-babc-4977-903c-89685a906d27 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:41.478767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d08babec-babc-4977-903c-89685a906d27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:41.549823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 646/1297] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:41 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:41.819535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75802259-dfb1-4e0f-96a1-acc1db0cc5c9 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:41.918676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75802259-dfb1-4e0f-96a1-acc1db0cc5c9 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:41.951602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:41.952234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:41.952511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:41.952773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:41.953335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:41.954676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:41.963510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:41.964785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:41.965976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 652/1298] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:44:41 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:41.970560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08babec-babc-4977-903c-89685a906d27 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:41.989435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:42.069031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08babec-babc-4977-903c-89685a906d27 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:42.101687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:42.102277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:42.102551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:42.102819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:42.103440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] RBAC: Adding query filter params (name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:42.103737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:42.105126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:42.110017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 647/1299] 10.222.0.42 () {56 vars in 1116 bytes} [Fri Sep 16 01:44:41 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:42.116675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:42.147288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:42.213238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaf205b-414d-4e7f-8a24-b021c5444858 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:42.246236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42fa8302-d909-46d0-b344-c1743c462adf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:42.246822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fa8302-d909-46d0-b344-c1743c462adf None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:42.247095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fa8302-d909-46d0-b344-c1743c462adf None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:42.247358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fa8302-d909-46d0-b344-c1743c462adf None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:42.248077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fa8302-d909-46d0-b344-c1743c462adf None admin] RBAC: Adding query filter params (interface=public, service_id=c1f8187eeac546b09468b4b95e3e4289, region_id=RegionOne) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:42.248383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fa8302-d909-46d0-b344-c1743c462adf None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:42.249721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42fa8302-d909-46d0-b344-c1743c462adf None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:42.254589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 653/1300] 10.222.0.42 () {56 vars in 1251 bytes} [Fri Sep 16 01:44:42 2022] GET /identity/v3/endpoints?service_id=c1f8187eeac546b09468b4b95e3e4289&interface=public®ion_id=RegionOne => generated 194 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:42.260463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:42.339297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:42.356981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0308e163-8652-4f37-84f6-19fb83dda9ad None admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:42.391682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcc8969-e183-47fd-9224-79c9c2bd9329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:42.392293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcc8969-e183-47fd-9224-79c9c2bd9329 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:42.392571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcc8969-e183-47fd-9224-79c9c2bd9329 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:42.392837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcc8969-e183-47fd-9224-79c9c2bd9329 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:42.393424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcc8969-e183-47fd-9224-79c9c2bd9329 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:42.395293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcc8969-e183-47fd-9224-79c9c2bd9329 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:42.401815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 648/1301] 10.222.0.42 () {56 vars in 1087 bytes} [Fri Sep 16 01:44:42 2022] GET /identity/v3/services => generated 2191 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:44:44.523211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc4fa8-5ad9-4f8f-883c-6427dcceeafa None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:44.523772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc4fa8-5ad9-4f8f-883c-6427dcceeafa None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:44.523772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fc4fa8-5ad9-4f8f-883c-6427dcceeafa None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:44.525120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 654/1302] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:44:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:44:44.530302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11932d3-fe5b-45cc-8110-f7e47e719b5b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:44.530710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11932d3-fe5b-45cc-8110-f7e47e719b5b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:44.530839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11932d3-fe5b-45cc-8110-f7e47e719b5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:44.549387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c11932d3-fe5b-45cc-8110-f7e47e719b5b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:44:44.550554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c11932d3-fe5b-45cc-8110-f7e47e719b5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:44:44.616254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 649/1303] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:44:44 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 16 01:44:44.883854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc4fa8-5ad9-4f8f-883c-6427dcceeafa None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:44.977069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:44.983881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fc4fa8-5ad9-4f8f-883c-6427dcceeafa None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:45.016830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:45.017443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:45.017722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:45.017988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:45.018558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:45.020043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 16 01:44:45.028607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 16 01:44:45.029773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.get_or_create( Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application with Lock( Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self._enter() Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application return self.creator() Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application created_value = creator( Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: ERROR keystone.server.flask.application Sep 16 01:44:45.030614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 655/1304] 10.222.0.42 () {56 vars in 1117 bytes} [Fri Sep 16 01:44:44 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 16 01:44:45.036108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11932d3-fe5b-45cc-8110-f7e47e719b5b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:45.109149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:45.132126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11932d3-fe5b-45cc-8110-f7e47e719b5b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:45.164912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698b9d8a-69d3-4de7-b376-07ed099a02f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:45.165497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b9d8a-69d3-4de7-b376-07ed099a02f8 None admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:45.165774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b9d8a-69d3-4de7-b376-07ed099a02f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:45.166041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b9d8a-69d3-4de7-b376-07ed099a02f8 None admin] PATH_INFO: `/v3/services` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:45.166661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698b9d8a-69d3-4de7-b376-07ed099a02f8 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:44:45.166954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698b9d8a-69d3-4de7-b376-07ed099a02f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:45.168336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698b9d8a-69d3-4de7-b376-07ed099a02f8 None admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:45.173215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 650/1305] 10.222.0.42 () {56 vars in 1116 bytes} [Fri Sep 16 01:44:45 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:44:45.179381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:44:45.277175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01f9033-faeb-486c-b9e0-ecf003823ac3 None admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:44:45.310190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4401a089-86eb-41b0-b7f1-85d54c06d247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:44:45.310779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4401a089-86eb-41b0-b7f1-85d54c06d247 None admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:44:45.311051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4401a089-86eb-41b0-b7f1-85d54c06d247 None admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:44:45.311312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4401a089-86eb-41b0-b7f1-85d54c06d247 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:44:45.311973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4401a089-86eb-41b0-b7f1-85d54c06d247 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:44:45.313320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4401a089-86eb-41b0-b7f1-85d54c06d247 None admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:44:45.331853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 656/1306] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:44:45 2022] POST /identity/v3/endpoints => generated 331 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:44:55.170401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=30 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:44:55.327760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:45:49.247306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7958f4-61d4-4b70-9b40-4ef9ac09d23b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:49.247786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7958f4-61d4-4b70-9b40-4ef9ac09d23b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:49.247874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7958f4-61d4-4b70-9b40-4ef9ac09d23b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:49.253588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 651/1307] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:45:49 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:45:49.255001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a55ba-8b18-4fed-b95d-8df849b9ca36 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:49.255414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a55ba-8b18-4fed-b95d-8df849b9ca36 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:49.255591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf8a55ba-8b18-4fed-b95d-8df849b9ca36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:49.408527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-bf8a55ba-8b18-4fed-b95d-8df849b9ca36 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:45:49.410399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bf8a55ba-8b18-4fed-b95d-8df849b9ca36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:49.572764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 657/1308] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:45:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:45:49.875119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116edfb3-0796-4d51-97af-afae732e1ed6 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:49.875483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116edfb3-0796-4d51-97af-afae732e1ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:49.875663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116edfb3-0796-4d51-97af-afae732e1ed6 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:49.877709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 652/1309] 10.222.0.42 () {54 vars in 965 bytes} [Fri Sep 16 01:45:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:45:49.883827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee3dc21-9b5f-43a1-98b1-bf05fb8f907f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:49.884707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee3dc21-9b5f-43a1-98b1-bf05fb8f907f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:49.885267 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee3dc21-9b5f-43a1-98b1-bf05fb8f907f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:49.978183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-2ee3dc21-9b5f-43a1-98b1-bf05fb8f907f None None] MFA Rules not processed for user `6a343a260fa2415e84caf959933a6c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:45:49.979395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2ee3dc21-9b5f-43a1-98b1-bf05fb8f907f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:50.002251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:45:50.072191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 658/1310] 10.222.0.42 () {58 vars in 1062 bytes} [Fri Sep 16 01:45:49 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:45:50.082807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116edfb3-0796-4d51-97af-afae732e1ed6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:45:50.084847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-116edfb3-0796-4d51-97af-afae732e1ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:50.218828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116edfb3-0796-4d51-97af-afae732e1ed6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:45:50.302073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e81197-0787-4833-a208-6a4001d9a425 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:45:50.302747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e81197-0787-4833-a208-6a4001d9a425 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:50.302980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e81197-0787-4833-a208-6a4001d9a425 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:50.303188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e81197-0787-4833-a208-6a4001d9a425 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:50.304851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-69e81197-0787-4833-a208-6a4001d9a425 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:50.362947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:45:50.426978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e81197-0787-4833-a208-6a4001d9a425 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:45:50.428504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e81197-0787-4833-a208-6a4001d9a425 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:45:50.451801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 653/1311] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:45:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:45:53.014856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f484a-25d1-4d89-852e-89832e6f369c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:53.015418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f484a-25d1-4d89-852e-89832e6f369c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:53.015418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f484a-25d1-4d89-852e-89832e6f369c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:53.017278 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 659/1312] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:45:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:45:53.022464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad2dacb-74e5-4741-bea2-2e337a91ce09 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:53.022875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad2dacb-74e5-4741-bea2-2e337a91ce09 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:53.023010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad2dacb-74e5-4741-bea2-2e337a91ce09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:53.097717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3ad2dacb-74e5-4741-bea2-2e337a91ce09 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:45:53.098937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3ad2dacb-74e5-4741-bea2-2e337a91ce09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:53.195437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 654/1313] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:45:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:45:53.483967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120f484a-25d1-4d89-852e-89832e6f369c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:45:53.486127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-120f484a-25d1-4d89-852e-89832e6f369c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:53.591298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120f484a-25d1-4d89-852e-89832e6f369c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:45:53.662053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:45:53.664765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55871e7-f6a6-4e57-a973-06d69651e9cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:45:53.665576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55871e7-f6a6-4e57-a973-06d69651e9cf service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:53.665899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55871e7-f6a6-4e57-a973-06d69651e9cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:53.666333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55871e7-f6a6-4e57-a973-06d69651e9cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:53.668167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d55871e7-f6a6-4e57-a973-06d69651e9cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:53.776957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55871e7-f6a6-4e57-a973-06d69651e9cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:45:53.778289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55871e7-f6a6-4e57-a973-06d69651e9cf service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:45:53.796603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 660/1314] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:45:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:45:56.244956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d61c2b-ed59-4ceb-b71d-1a849971d39b None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:56.245477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d61c2b-ed59-4ceb-b71d-1a849971d39b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:56.245477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d61c2b-ed59-4ceb-b71d-1a849971d39b None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:56.246868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 655/1315] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:45:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:45:56.252209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab6805f-794b-4cb6-9f9e-31aff1fd69fd None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:56.252610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab6805f-794b-4cb6-9f9e-31aff1fd69fd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:56.252758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab6805f-794b-4cb6-9f9e-31aff1fd69fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:56.318737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-fab6805f-794b-4cb6-9f9e-31aff1fd69fd None None] MFA Rules not processed for user `a5d7b215b8b94e8d8b6ec1b2b1dc9797`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:45:56.319959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fab6805f-794b-4cb6-9f9e-31aff1fd69fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:56.370353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:45:56.416172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 661/1316] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:45:56 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:45:56.702425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d61c2b-ed59-4ceb-b71d-1a849971d39b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:45:56.703975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-79d61c2b-ed59-4ceb-b71d-1a849971d39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:56.713746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:45:56.808395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d61c2b-ed59-4ceb-b71d-1a849971d39b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:45:56.880456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b1eb85-6130-432b-8bc6-83ad81d4d2f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:45:56.881048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b1eb85-6130-432b-8bc6-83ad81d4d2f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:56.881321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b1eb85-6130-432b-8bc6-83ad81d4d2f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:56.881587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b1eb85-6130-432b-8bc6-83ad81d4d2f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:56.883159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-11b1eb85-6130-432b-8bc6-83ad81d4d2f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:56.943391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:45:56.989764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b1eb85-6130-432b-8bc6-83ad81d4d2f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:45:56.991111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b1eb85-6130-432b-8bc6-83ad81d4d2f9 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:45:57.009160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 656/1317] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:45:56 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:45:59.705652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80f4574-8f20-4500-b3c5-7eabc8ceff4d None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:59.706191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80f4574-8f20-4500-b3c5-7eabc8ceff4d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:59.706191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80f4574-8f20-4500-b3c5-7eabc8ceff4d None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:59.707567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 662/1318] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:45:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:45:59.712928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9239cbee-f010-4e7c-afd6-f9d153d04433 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:45:59.713376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9239cbee-f010-4e7c-afd6-f9d153d04433 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:45:59.713471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9239cbee-f010-4e7c-afd6-f9d153d04433 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:45:59.779361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9239cbee-f010-4e7c-afd6-f9d153d04433 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:45:59.780591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9239cbee-f010-4e7c-afd6-f9d153d04433 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:45:59.884202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 657/1319] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:45:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:00.172369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80f4574-8f20-4500-b3c5-7eabc8ceff4d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:00.173941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c80f4574-8f20-4500-b3c5-7eabc8ceff4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:00.276821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80f4574-8f20-4500-b3c5-7eabc8ceff4d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:00.348170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8b900a-b61f-4edd-b572-2da02291ec96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:00.348762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8b900a-b61f-4edd-b572-2da02291ec96 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:00.349042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8b900a-b61f-4edd-b572-2da02291ec96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:00.349312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8b900a-b61f-4edd-b572-2da02291ec96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:00.350873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2b8b900a-b61f-4edd-b572-2da02291ec96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:00.357583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:00.460795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8b900a-b61f-4edd-b572-2da02291ec96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:00.462145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8b900a-b61f-4edd-b572-2da02291ec96 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:00.480237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 663/1320] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:00 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:03.302615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f0ca3d-43c9-41f0-bf12-b042cf859abb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:03.303178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f0ca3d-43c9-41f0-bf12-b042cf859abb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:03.303178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f0ca3d-43c9-41f0-bf12-b042cf859abb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:03.304765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 658/1321] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:03.310713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f822fc-4f61-40f3-9248-0b1c2118f7f7 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:03.311242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f822fc-4f61-40f3-9248-0b1c2118f7f7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:03.311242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f822fc-4f61-40f3-9248-0b1c2118f7f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:03.386730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-95f822fc-4f61-40f3-9248-0b1c2118f7f7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:03.387999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-95f822fc-4f61-40f3-9248-0b1c2118f7f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:03.453019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:03.497046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 664/1322] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:03.786175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f0ca3d-43c9-41f0-bf12-b042cf859abb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:03.787766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-71f0ca3d-43c9-41f0-bf12-b042cf859abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:03.809355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:03.894616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f0ca3d-43c9-41f0-bf12-b042cf859abb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:03.967880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b69c345-53d2-484f-8e00-9d708c938ab1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:03.968497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b69c345-53d2-484f-8e00-9d708c938ab1 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:03.968771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b69c345-53d2-484f-8e00-9d708c938ab1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:03.969037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b69c345-53d2-484f-8e00-9d708c938ab1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:03.970655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0b69c345-53d2-484f-8e00-9d708c938ab1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:04.042010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:04.082852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b69c345-53d2-484f-8e00-9d708c938ab1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:04.084312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b69c345-53d2-484f-8e00-9d708c938ab1 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:04.102058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 659/1323] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:06.340012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9579db-7a20-400a-b8d3-f858af517843 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:06.340563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9579db-7a20-400a-b8d3-f858af517843 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:06.340563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9579db-7a20-400a-b8d3-f858af517843 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:06.341873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 665/1324] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:06.347581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61404af1-4a3f-4f3c-a253-0f08d46d3593 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:06.348036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61404af1-4a3f-4f3c-a253-0f08d46d3593 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:06.348158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61404af1-4a3f-4f3c-a253-0f08d46d3593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:06.429954 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-61404af1-4a3f-4f3c-a253-0f08d46d3593 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:06.431858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-61404af1-4a3f-4f3c-a253-0f08d46d3593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:06.530519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 660/1325] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:06.814192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9579db-7a20-400a-b8d3-f858af517843 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:06.815698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-eb9579db-7a20-400a-b8d3-f858af517843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:06.926754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9579db-7a20-400a-b8d3-f858af517843 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:07.002917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d691924e-8161-405b-9082-cd6b519f8f3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:07.003519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d691924e-8161-405b-9082-cd6b519f8f3c service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:07.003792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d691924e-8161-405b-9082-cd6b519f8f3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:07.004088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d691924e-8161-405b-9082-cd6b519f8f3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:07.005663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d691924e-8161-405b-9082-cd6b519f8f3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:07.024166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:07.119974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d691924e-8161-405b-9082-cd6b519f8f3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:07.121431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d691924e-8161-405b-9082-cd6b519f8f3c service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:07.140052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 666/1326] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:46:09.366483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27f15d-b357-4d5c-8f6b-01e6ee075616 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:09.367008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27f15d-b357-4d5c-8f6b-01e6ee075616 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:09.367008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27f15d-b357-4d5c-8f6b-01e6ee075616 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:09.372447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 661/1327] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:09.374472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7f7499-bcdb-4dee-8957-4cad570b8a4c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:09.374894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7f7499-bcdb-4dee-8957-4cad570b8a4c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:09.375007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7f7499-bcdb-4dee-8957-4cad570b8a4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:09.447834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-3e7f7499-bcdb-4dee-8957-4cad570b8a4c None None] MFA Rules not processed for user `a5d7b215b8b94e8d8b6ec1b2b1dc9797`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:09.449091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3e7f7499-bcdb-4dee-8957-4cad570b8a4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:09.515713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:09.548436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 667/1328] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:09 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:09.846792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd6330-c980-40f7-8b96-10b4535bb176 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:09.847076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd6330-c980-40f7-8b96-10b4535bb176 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:09.847351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dd6330-c980-40f7-8b96-10b4535bb176 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:09.848721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 662/1329] 10.222.0.42 () {54 vars in 965 bytes} [Fri Sep 16 01:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:09.854645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcb7890-a11e-4117-bb4e-7c88fca0162e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:09.855015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcb7890-a11e-4117-bb4e-7c88fca0162e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:09.855353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcb7890-a11e-4117-bb4e-7c88fca0162e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:09.932312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4fcb7890-a11e-4117-bb4e-7c88fca0162e None None] MFA Rules not processed for user `6a343a260fa2415e84caf959933a6c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:09.933428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4fcb7890-a11e-4117-bb4e-7c88fca0162e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:10.013385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 668/1330] 10.222.0.42 () {58 vars in 1062 bytes} [Fri Sep 16 01:46:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:10.022525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd6330-c980-40f7-8b96-10b4535bb176 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:10.024061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-91dd6330-c980-40f7-8b96-10b4535bb176 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:10.055740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:10.127293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dd6330-c980-40f7-8b96-10b4535bb176 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:10.202584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fcbf22-813e-40c5-8c6d-ab98c8d7065b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:10.203100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fcbf22-813e-40c5-8c6d-ab98c8d7065b service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:10.203881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fcbf22-813e-40c5-8c6d-ab98c8d7065b service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:10.204177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fcbf22-813e-40c5-8c6d-ab98c8d7065b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:10.205951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-44fcbf22-813e-40c5-8c6d-ab98c8d7065b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:10.288090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:10.321996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fcbf22-813e-40c5-8c6d-ab98c8d7065b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:10.323439 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fcbf22-813e-40c5-8c6d-ab98c8d7065b service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:10.342254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 663/1331] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:10 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:13.124301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e569eb-2c09-401b-8ab7-5dfacd63c798 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:13.124831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e569eb-2c09-401b-8ab7-5dfacd63c798 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:13.124831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e569eb-2c09-401b-8ab7-5dfacd63c798 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:13.126192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 669/1332] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:13.131347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01df706-ce26-4b96-bad4-c4da1b888a8a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:13.131801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01df706-ce26-4b96-bad4-c4da1b888a8a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:13.131902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01df706-ce26-4b96-bad4-c4da1b888a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:13.197163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a01df706-ce26-4b96-bad4-c4da1b888a8a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:13.198363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a01df706-ce26-4b96-bad4-c4da1b888a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:13.292872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 664/1333] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:13.581380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e569eb-2c09-401b-8ab7-5dfacd63c798 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:13.582926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-20e569eb-2c09-401b-8ab7-5dfacd63c798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:13.651998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:13.685400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e569eb-2c09-401b-8ab7-5dfacd63c798 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:13.757953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5b79e0-9c13-4630-807e-4dd7a7f77a1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:13.758551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b79e0-9c13-4630-807e-4dd7a7f77a1f service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:13.759094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b79e0-9c13-4630-807e-4dd7a7f77a1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:13.759360 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5b79e0-9c13-4630-807e-4dd7a7f77a1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:13.760969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ed5b79e0-9c13-4630-807e-4dd7a7f77a1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:13.870424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5b79e0-9c13-4630-807e-4dd7a7f77a1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:13.871850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5b79e0-9c13-4630-807e-4dd7a7f77a1f service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:13.882833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:13.891925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 670/1334] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:16.094224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ed02c-bf4c-41c9-9b52-6a3a4051dd6d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:16.094989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ed02c-bf4c-41c9-9b52-6a3a4051dd6d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:16.094989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939ed02c-bf4c-41c9-9b52-6a3a4051dd6d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:16.096261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 665/1335] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:16.101574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985aaecc-6ac7-4274-8aaf-f6b09fc7a8fa None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:16.101992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985aaecc-6ac7-4274-8aaf-f6b09fc7a8fa None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:16.102116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985aaecc-6ac7-4274-8aaf-f6b09fc7a8fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:16.166861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-985aaecc-6ac7-4274-8aaf-f6b09fc7a8fa None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:16.168084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-985aaecc-6ac7-4274-8aaf-f6b09fc7a8fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:16.264848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 671/1336] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:16.549131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ed02c-bf4c-41c9-9b52-6a3a4051dd6d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:16.550668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-939ed02c-bf4c-41c9-9b52-6a3a4051dd6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:16.589974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:16.653957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939ed02c-bf4c-41c9-9b52-6a3a4051dd6d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:16.727066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1176c99-cf6f-4ae5-8896-4c3334a7c393 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:16.727655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1176c99-cf6f-4ae5-8896-4c3334a7c393 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:16.727922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1176c99-cf6f-4ae5-8896-4c3334a7c393 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:16.728212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1176c99-cf6f-4ae5-8896-4c3334a7c393 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:16.729768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b1176c99-cf6f-4ae5-8896-4c3334a7c393 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:16.812526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:16.838158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1176c99-cf6f-4ae5-8896-4c3334a7c393 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:16.839593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1176c99-cf6f-4ae5-8896-4c3334a7c393 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:16.858515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 666/1337] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:19.336724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94662472-2c0c-4729-89bf-fe2dbee2316c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:19.337225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94662472-2c0c-4729-89bf-fe2dbee2316c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:19.337225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94662472-2c0c-4729-89bf-fe2dbee2316c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:19.338504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 672/1338] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:46:19.343762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ab5c1-f3c0-49d5-97f7-8b88138959fe None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:19.344273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ab5c1-f3c0-49d5-97f7-8b88138959fe None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:19.344369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8ab5c1-f3c0-49d5-97f7-8b88138959fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:19.417165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-aa8ab5c1-f3c0-49d5-97f7-8b88138959fe None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:19.418947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-aa8ab5c1-f3c0-49d5-97f7-8b88138959fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:19.521773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 667/1339] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:19.804072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94662472-2c0c-4729-89bf-fe2dbee2316c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:19.805623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-94662472-2c0c-4729-89bf-fe2dbee2316c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:19.881384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:19.909596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94662472-2c0c-4729-89bf-fe2dbee2316c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:19.982351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92660c42-01be-4814-ac1d-eeb621cd50bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:19.982947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92660c42-01be-4814-ac1d-eeb621cd50bc service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:19.983221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92660c42-01be-4814-ac1d-eeb621cd50bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:19.983485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92660c42-01be-4814-ac1d-eeb621cd50bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:19.985124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-92660c42-01be-4814-ac1d-eeb621cd50bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:20.096489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92660c42-01be-4814-ac1d-eeb621cd50bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:20.097842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92660c42-01be-4814-ac1d-eeb621cd50bc service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:20.116051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 673/1340] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:24.347450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0cd57-019e-430d-a597-c024b630b296 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:24.347845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0cd57-019e-430d-a597-c024b630b296 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:24.348112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0cd57-019e-430d-a597-c024b630b296 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:24.352331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 668/1341] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:24.354784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f96ad-ea93-41cd-863b-ae3c84575e56 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:24.355191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f96ad-ea93-41cd-863b-ae3c84575e56 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:24.355353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45f96ad-ea93-41cd-863b-ae3c84575e56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:24.362094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:24.432742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e45f96ad-ea93-41cd-863b-ae3c84575e56 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:24.434007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e45f96ad-ea93-41cd-863b-ae3c84575e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:24.542153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 674/1342] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:24.829473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0cd57-019e-430d-a597-c024b630b296 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:24.831017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b3f0cd57-019e-430d-a597-c024b630b296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:24.881623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:24.939287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0cd57-019e-430d-a597-c024b630b296 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:25.014783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5f9dfa-64b6-461b-9999-ec85f91dab64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:25.016012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5f9dfa-64b6-461b-9999-ec85f91dab64 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:25.016293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5f9dfa-64b6-461b-9999-ec85f91dab64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:25.016559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5f9dfa-64b6-461b-9999-ec85f91dab64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:25.018135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dc5f9dfa-64b6-461b-9999-ec85f91dab64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:25.114679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:25.132255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5f9dfa-64b6-461b-9999-ec85f91dab64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:25.133940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc5f9dfa-64b6-461b-9999-ec85f91dab64 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:25.153117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 669/1343] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:27.971919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2c489-2272-478c-a382-da263682c043 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:27.972524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2c489-2272-478c-a382-da263682c043 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:27.972524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa2c489-2272-478c-a382-da263682c043 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:27.973827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 675/1344] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:27.979111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edef825c-bdf9-4094-af57-6d8a15b9868e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:27.979503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edef825c-bdf9-4094-af57-6d8a15b9868e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:27.979791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edef825c-bdf9-4094-af57-6d8a15b9868e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:28.046221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-edef825c-bdf9-4094-af57-6d8a15b9868e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:28.047426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-edef825c-bdf9-4094-af57-6d8a15b9868e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:28.143926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 670/1345] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:27 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:28.430925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2c489-2272-478c-a382-da263682c043 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:28.432463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8fa2c489-2272-478c-a382-da263682c043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:28.527752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:28.545448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa2c489-2272-478c-a382-da263682c043 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:28.619128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990df834-f3b3-4bb1-86d6-59e38c8d8ca7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:28.619730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990df834-f3b3-4bb1-86d6-59e38c8d8ca7 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:28.620035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990df834-f3b3-4bb1-86d6-59e38c8d8ca7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:28.620334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990df834-f3b3-4bb1-86d6-59e38c8d8ca7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:28.621922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-990df834-f3b3-4bb1-86d6-59e38c8d8ca7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:28.731668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990df834-f3b3-4bb1-86d6-59e38c8d8ca7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:28.733228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990df834-f3b3-4bb1-86d6-59e38c8d8ca7 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:28.752073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 676/1346] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:28 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:33.240379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d1d4ec-da93-400c-b38e-f9e4698b7e5a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:33.240856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d1d4ec-da93-400c-b38e-f9e4698b7e5a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:33.240942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d1d4ec-da93-400c-b38e-f9e4698b7e5a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:33.242252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 671/1347] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:33.247728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843146f6-51b6-4afb-87b2-694949b53aae None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:33.248439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843146f6-51b6-4afb-87b2-694949b53aae None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:33.248439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843146f6-51b6-4afb-87b2-694949b53aae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:33.264278 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:33.325175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-843146f6-51b6-4afb-87b2-694949b53aae None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:33.326507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-843146f6-51b6-4afb-87b2-694949b53aae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:33.429592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 677/1348] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:33.713290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d1d4ec-da93-400c-b38e-f9e4698b7e5a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:33.714834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-81d1d4ec-da93-400c-b38e-f9e4698b7e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:33.786394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:33.832723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d1d4ec-da93-400c-b38e-f9e4698b7e5a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:33.904661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f2c85-98ac-4478-bd2c-53abb4f74bee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:33.905252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f2c85-98ac-4478-bd2c-53abb4f74bee service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:33.906192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f2c85-98ac-4478-bd2c-53abb4f74bee service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:33.906463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f2c85-98ac-4478-bd2c-53abb4f74bee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:33.908059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c94f2c85-98ac-4478-bd2c-53abb4f74bee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:34.018776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:34.022947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f2c85-98ac-4478-bd2c-53abb4f74bee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:34.024409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f2c85-98ac-4478-bd2c-53abb4f74bee service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:34.042372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 672/1349] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:38.967605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06a30c8-7db9-43db-8bc3-9e5a9bb6fac4 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:38.968218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06a30c8-7db9-43db-8bc3-9e5a9bb6fac4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:38.968218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06a30c8-7db9-43db-8bc3-9e5a9bb6fac4 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:38.969767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 678/1350] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:46:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:38.975787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10345bf4-c10e-4909-8088-d7b46695d980 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:38.976246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10345bf4-c10e-4909-8088-d7b46695d980 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:38.976344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10345bf4-c10e-4909-8088-d7b46695d980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:39.120759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-10345bf4-c10e-4909-8088-d7b46695d980 None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:39.121862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-10345bf4-c10e-4909-8088-d7b46695d980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:39.239191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 673/1351] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:46:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:46:43.421632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3600042-f37e-4b30-9c85-9bac7ac1ee28 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:43.422335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3600042-f37e-4b30-9c85-9bac7ac1ee28 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:43.422572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3600042-f37e-4b30-9c85-9bac7ac1ee28 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:43.424341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 679/1352] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:46:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:43.429214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=59 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:43.430710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820a6fef-d76f-4f26-b216-1d30fedb9aeb None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:43.431498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820a6fef-d76f-4f26-b216-1d30fedb9aeb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:43.431769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820a6fef-d76f-4f26-b216-1d30fedb9aeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:43.491268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:43.516952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-820a6fef-d76f-4f26-b216-1d30fedb9aeb None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:43.518146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-820a6fef-d76f-4f26-b216-1d30fedb9aeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:43.628424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 674/1353] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:46:43 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:45.032201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d96233-4b03-4350-b8d7-14f88b052582 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:45.032771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d96233-4b03-4350-b8d7-14f88b052582 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:45.032771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d96233-4b03-4350-b8d7-14f88b052582 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:45.034102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 680/1354] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:45.039144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072d64d-8b71-4ae2-95bc-7ee9f4b6b524 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:45.039562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072d64d-8b71-4ae2-95bc-7ee9f4b6b524 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:45.039684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1072d64d-8b71-4ae2-95bc-7ee9f4b6b524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:45.118077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1072d64d-8b71-4ae2-95bc-7ee9f4b6b524 None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:45.119281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1072d64d-8b71-4ae2-95bc-7ee9f4b6b524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:45.161805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:45.234704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 675/1355] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:46:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:46:48.807406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7eb500-0b75-40b9-8b46-0340f9afa29c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:48.808563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7eb500-0b75-40b9-8b46-0340f9afa29c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:48.808840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7eb500-0b75-40b9-8b46-0340f9afa29c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:48.810734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 681/1356] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:48.816970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b4d7be-c864-4507-a798-44fd3726d1b6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:48.817382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b4d7be-c864-4507-a798-44fd3726d1b6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:48.817506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b4d7be-c864-4507-a798-44fd3726d1b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:48.891444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-60b4d7be-c864-4507-a798-44fd3726d1b6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:48.892646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-60b4d7be-c864-4507-a798-44fd3726d1b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:49.009409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 676/1357] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:49.308292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a38f0e-32b3-43f2-a11a-118095c0c37e None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:49.309021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a38f0e-32b3-43f2-a11a-118095c0c37e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:49.309510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a38f0e-32b3-43f2-a11a-118095c0c37e None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:49.311273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 682/1358] 10.222.0.42 () {54 vars in 962 bytes} [Fri Sep 16 01:46:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:49.321352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d8998-4661-49b4-a969-a6deea97d4f6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:49.323440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d8998-4661-49b4-a969-a6deea97d4f6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:49.324724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9d8998-4661-49b4-a969-a6deea97d4f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:49.345669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:49.410776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ab9d8998-4661-49b4-a969-a6deea97d4f6 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:49.412086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ab9d8998-4661-49b4-a969-a6deea97d4f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:49.544710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 677/1359] 10.222.0.42 () {58 vars in 1059 bytes} [Fri Sep 16 01:46:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:46:49.553358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a38f0e-32b3-43f2-a11a-118095c0c37e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:49.555127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-80a38f0e-32b3-43f2-a11a-118095c0c37e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:49.720255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a38f0e-32b3-43f2-a11a-118095c0c37e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:49.800047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:49.848836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae89dc4-5ea7-4a6b-a861-1482bdc2ee54 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:49.849512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae89dc4-5ea7-4a6b-a861-1482bdc2ee54 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:49.849758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae89dc4-5ea7-4a6b-a861-1482bdc2ee54 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:49.849962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae89dc4-5ea7-4a6b-a861-1482bdc2ee54 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:49.851519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0ae89dc4-5ea7-4a6b-a861-1482bdc2ee54 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:49.965752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae89dc4-5ea7-4a6b-a861-1482bdc2ee54 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:49.968062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae89dc4-5ea7-4a6b-a861-1482bdc2ee54 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:49.987873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 683/1360] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:46:52.438417 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec708a-afbb-43d0-8007-39c0377c4f6f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:52.438918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec708a-afbb-43d0-8007-39c0377c4f6f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:52.438918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ec708a-afbb-43d0-8007-39c0377c4f6f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:52.440726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 678/1361] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:52.445677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5782ccf4-045f-47af-91ff-e326b76073a0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:52.446104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5782ccf4-045f-47af-91ff-e326b76073a0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:52.446355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5782ccf4-045f-47af-91ff-e326b76073a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:52.526839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5782ccf4-045f-47af-91ff-e326b76073a0 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:52.528162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5782ccf4-045f-47af-91ff-e326b76073a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:52.569668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:52.636212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 684/1362] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:52.936839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec708a-afbb-43d0-8007-39c0377c4f6f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:52.938566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-97ec708a-afbb-43d0-8007-39c0377c4f6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:53.019903 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:53.136687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ec708a-afbb-43d0-8007-39c0377c4f6f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:53.273347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e51b8eb-055c-4d3b-95fe-ca3080bf803e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:53.273879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51b8eb-055c-4d3b-95fe-ca3080bf803e service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:53.274159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51b8eb-055c-4d3b-95fe-ca3080bf803e service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:53.274430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51b8eb-055c-4d3b-95fe-ca3080bf803e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:53.276096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5e51b8eb-055c-4d3b-95fe-ca3080bf803e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:53.303003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:53.376302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51b8eb-055c-4d3b-95fe-ca3080bf803e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:53.377920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51b8eb-055c-4d3b-95fe-ca3080bf803e service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:53.396872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 679/1363] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:55.930342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc77d17-73c8-45a3-9532-ce5a971a8e2c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:55.930903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc77d17-73c8-45a3-9532-ce5a971a8e2c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:55.930903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc77d17-73c8-45a3-9532-ce5a971a8e2c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:55.932267 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 685/1364] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:55.937956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43486532-59d0-49f0-9c37-d7777d1fdfce None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:55.938394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43486532-59d0-49f0-9c37-d7777d1fdfce None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:55.938523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43486532-59d0-49f0-9c37-d7777d1fdfce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:56.009024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-43486532-59d0-49f0-9c37-d7777d1fdfce None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:56.010379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-43486532-59d0-49f0-9c37-d7777d1fdfce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:56.114836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:56.128095 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 680/1365] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:56.411466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce199e0-aaab-4736-8dc5-3a14ce70f2f4 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:56.411892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce199e0-aaab-4736-8dc5-3a14ce70f2f4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:56.412338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce199e0-aaab-4736-8dc5-3a14ce70f2f4 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:56.413943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 686/1366] 10.222.0.42 () {54 vars in 962 bytes} [Fri Sep 16 01:46:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:56.419289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd64458-1617-4c2f-90ed-50b58bdfbca4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:56.419683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd64458-1617-4c2f-90ed-50b58bdfbca4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:56.420738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd64458-1617-4c2f-90ed-50b58bdfbca4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:56.503225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-cfd64458-1617-4c2f-90ed-50b58bdfbca4 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:56.504569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cfd64458-1617-4c2f-90ed-50b58bdfbca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:56.629731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 681/1367] 10.222.0.42 () {58 vars in 1059 bytes} [Fri Sep 16 01:46:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:46:56.638924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce199e0-aaab-4736-8dc5-3a14ce70f2f4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:56.640619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2ce199e0-aaab-4736-8dc5-3a14ce70f2f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:56.800637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce199e0-aaab-4736-8dc5-3a14ce70f2f4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:56.825658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:56.933442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab72e5c4-1d28-47d9-aeb6-487ec80f5d05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:56.934128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab72e5c4-1d28-47d9-aeb6-487ec80f5d05 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:56.934468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab72e5c4-1d28-47d9-aeb6-487ec80f5d05 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:56.934804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab72e5c4-1d28-47d9-aeb6-487ec80f5d05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:56.937162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ab72e5c4-1d28-47d9-aeb6-487ec80f5d05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:57.040887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab72e5c4-1d28-47d9-aeb6-487ec80f5d05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:57.042874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab72e5c4-1d28-47d9-aeb6-487ec80f5d05 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:57.069249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 687/1368] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:57.149217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4cb262-6073-4599-a615-cc4041f94327 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:57.149765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4cb262-6073-4599-a615-cc4041f94327 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:57.149765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4cb262-6073-4599-a615-cc4041f94327 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:57.151066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 682/1369] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:57.156843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf29b65b-be28-4574-a99a-05fe631d5783 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:57.157138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf29b65b-be28-4574-a99a-05fe631d5783 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:57.157441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf29b65b-be28-4574-a99a-05fe631d5783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:57.197120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:57.246218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-cf29b65b-be28-4574-a99a-05fe631d5783 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:57.247653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cf29b65b-be28-4574-a99a-05fe631d5783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:57.371681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 688/1370] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:57.716580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f716f-5f73-4412-bcaa-54aee3f39494 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:57.716580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f716f-5f73-4412-bcaa-54aee3f39494 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:57.716580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9f716f-5f73-4412-bcaa-54aee3f39494 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:57.721645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 683/1371] 10.222.0.42 () {54 vars in 964 bytes} [Fri Sep 16 01:46:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:57.723570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253a20c-5e7a-44bc-bac1-23094ef9ea03 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:57.723860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253a20c-5e7a-44bc-bac1-23094ef9ea03 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:57.724233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253a20c-5e7a-44bc-bac1-23094ef9ea03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:57.811733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7253a20c-5e7a-44bc-bac1-23094ef9ea03 None None] MFA Rules not processed for user `d7b63222dc2846f5a42208877b0de123`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:57.813494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7253a20c-5e7a-44bc-bac1-23094ef9ea03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:57.863383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:57.897136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 689/1372] 10.222.0.42 () {58 vars in 1061 bytes} [Fri Sep 16 01:46:57 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:57.904818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9f716f-5f73-4412-bcaa-54aee3f39494 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:57.906553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cc9f716f-5f73-4412-bcaa-54aee3f39494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:57.959957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:58.021215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9f716f-5f73-4412-bcaa-54aee3f39494 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:46:58.121254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b92af11-150a-49d7-8c1e-73d303ac2d15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:46:58.121760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92af11-150a-49d7-8c1e-73d303ac2d15 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:58.122064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92af11-150a-49d7-8c1e-73d303ac2d15 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:58.122358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b92af11-150a-49d7-8c1e-73d303ac2d15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:58.124608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1b92af11-150a-49d7-8c1e-73d303ac2d15 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:58.290604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:46:58.293279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b92af11-150a-49d7-8c1e-73d303ac2d15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:46:58.294898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b92af11-150a-49d7-8c1e-73d303ac2d15 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:46:58.320869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 684/1373] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:46:59.360918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d53f3d2-7db6-47b8-b93f-e775c3a0bbe2 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:59.361450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d53f3d2-7db6-47b8-b93f-e775c3a0bbe2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:59.361450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d53f3d2-7db6-47b8-b93f-e775c3a0bbe2 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:59.362770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 690/1374] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:46:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:46:59.368185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac8169-c775-45df-b244-4356ee3a664a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:46:59.368641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac8169-c775-45df-b244-4356ee3a664a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:46:59.368763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ac8169-c775-45df-b244-4356ee3a664a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:46:59.438187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-01ac8169-c775-45df-b244-4356ee3a664a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:46:59.439409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-01ac8169-c775-45df-b244-4356ee3a664a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:59.537432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 685/1375] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:46:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:46:59.830633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d53f3d2-7db6-47b8-b93f-e775c3a0bbe2 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:46:59.832196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2d53f3d2-7db6-47b8-b93f-e775c3a0bbe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:46:59.997719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d53f3d2-7db6-47b8-b93f-e775c3a0bbe2 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:00.053049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:00.118710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ea9bd2-25df-4cca-b844-f04cb2418370 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:00.119323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea9bd2-25df-4cca-b844-f04cb2418370 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:00.119603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea9bd2-25df-4cca-b844-f04cb2418370 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:00.119874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ea9bd2-25df-4cca-b844-f04cb2418370 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:00.121504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-94ea9bd2-25df-4cca-b844-f04cb2418370 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:00.216223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ea9bd2-25df-4cca-b844-f04cb2418370 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:00.218545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ea9bd2-25df-4cca-b844-f04cb2418370 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:00.237900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 691/1376] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:46:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:02.460694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6f271a-c14d-47dd-bc9f-79e974b91fe4 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:02.461251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6f271a-c14d-47dd-bc9f-79e974b91fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:02.461251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6f271a-c14d-47dd-bc9f-79e974b91fe4 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:02.462904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 686/1377] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:02.468513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec2693d-13cf-46e1-a0a1-5530119264a4 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:02.469289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec2693d-13cf-46e1-a0a1-5530119264a4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:02.469568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec2693d-13cf-46e1-a0a1-5530119264a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:02.539283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-9ec2693d-13cf-46e1-a0a1-5530119264a4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:02.540532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9ec2693d-13cf-46e1-a0a1-5530119264a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:02.548674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:02.648609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 692/1378] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:02.941384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6f271a-c14d-47dd-bc9f-79e974b91fe4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:02.942920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1c6f271a-c14d-47dd-bc9f-79e974b91fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:03.007175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:03.099125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6f271a-c14d-47dd-bc9f-79e974b91fe4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:03.224115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52bf52-7077-4d6f-8866-63ed60e16d9c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:03.224708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52bf52-7077-4d6f-8866-63ed60e16d9c service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:03.224990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52bf52-7077-4d6f-8866-63ed60e16d9c service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:03.225260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52bf52-7077-4d6f-8866-63ed60e16d9c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:03.226888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ce52bf52-7077-4d6f-8866-63ed60e16d9c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:03.247081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:03.324833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52bf52-7077-4d6f-8866-63ed60e16d9c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:03.327152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52bf52-7077-4d6f-8866-63ed60e16d9c service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:03.347209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 687/1379] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:05.586550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb5506-54e1-4581-82fe-574cf641168b None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:05.587050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb5506-54e1-4581-82fe-574cf641168b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:05.587050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb5506-54e1-4581-82fe-574cf641168b None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:05.588528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 693/1380] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:05.593773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44424c7-ba74-49bf-98dd-ff6fd8603f08 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:05.594187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44424c7-ba74-49bf-98dd-ff6fd8603f08 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:05.594356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44424c7-ba74-49bf-98dd-ff6fd8603f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:05.664896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c44424c7-ba74-49bf-98dd-ff6fd8603f08 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:05.666078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c44424c7-ba74-49bf-98dd-ff6fd8603f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:05.784263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:05.800551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 688/1381] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:06.133712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb5506-54e1-4581-82fe-574cf641168b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:06.135228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-22eb5506-54e1-4581-82fe-574cf641168b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:06.300295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:06.313619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb5506-54e1-4581-82fe-574cf641168b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:06.463658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0dc206-d915-44c8-957b-9e93a9a292ef service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:06.464328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dc206-d915-44c8-957b-9e93a9a292ef service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:06.464640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dc206-d915-44c8-957b-9e93a9a292ef service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:06.464963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0dc206-d915-44c8-957b-9e93a9a292ef service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:06.467525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ec0dc206-d915-44c8-957b-9e93a9a292ef service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:06.585273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0dc206-d915-44c8-957b-9e93a9a292ef service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:06.587309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0dc206-d915-44c8-957b-9e93a9a292ef service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:06.600872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:06.611789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 694/1382] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:08.856358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef7f957-bdb1-42cd-acf4-a8b07f5009c4 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:08.857109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef7f957-bdb1-42cd-acf4-a8b07f5009c4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:08.857109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef7f957-bdb1-42cd-acf4-a8b07f5009c4 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:08.858821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 689/1383] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:08.864639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42721f5-29a4-4b92-99c0-b155dabc2654 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:08.865154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42721f5-29a4-4b92-99c0-b155dabc2654 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:08.865252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42721f5-29a4-4b92-99c0-b155dabc2654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:08.950731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d42721f5-29a4-4b92-99c0-b155dabc2654 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:08.952462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d42721f5-29a4-4b92-99c0-b155dabc2654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:09.084414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 695/1384] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:47:09.378410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef7f957-bdb1-42cd-acf4-a8b07f5009c4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:09.380609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bef7f957-bdb1-42cd-acf4-a8b07f5009c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:09.561286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef7f957-bdb1-42cd-acf4-a8b07f5009c4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:09.655615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:09.709571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37db81e5-e0d4-491c-85d2-45d979d8d04b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:09.710170 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37db81e5-e0d4-491c-85d2-45d979d8d04b service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:09.710449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37db81e5-e0d4-491c-85d2-45d979d8d04b service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:09.710745 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37db81e5-e0d4-491c-85d2-45d979d8d04b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:09.712397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-37db81e5-e0d4-491c-85d2-45d979d8d04b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:09.817473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37db81e5-e0d4-491c-85d2-45d979d8d04b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:09.819119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37db81e5-e0d4-491c-85d2-45d979d8d04b service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:09.839235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 690/1385] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 462 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:12.219656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf2b52-744b-404d-ac4b-f2260c141407 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:12.220228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf2b52-744b-404d-ac4b-f2260c141407 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:12.220228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf2b52-744b-404d-ac4b-f2260c141407 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:12.221591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 696/1386] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:12.226991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d280a4a-a15d-44ba-ac55-5b582caa26f0 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:12.227405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d280a4a-a15d-44ba-ac55-5b582caa26f0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:12.227571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d280a4a-a15d-44ba-ac55-5b582caa26f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:12.295927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6d280a4a-a15d-44ba-ac55-5b582caa26f0 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:12.297167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6d280a4a-a15d-44ba-ac55-5b582caa26f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:12.331528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:12.396168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 691/1387] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:12.690375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bf2b52-744b-404d-ac4b-f2260c141407 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:12.691962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c1bf2b52-744b-404d-ac4b-f2260c141407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:12.769700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:12.842080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bf2b52-744b-404d-ac4b-f2260c141407 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:12.954757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21269e6-0190-4904-a89e-b843fb19b2e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:12.955279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21269e6-0190-4904-a89e-b843fb19b2e4 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:12.955559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21269e6-0190-4904-a89e-b843fb19b2e4 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:12.955827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21269e6-0190-4904-a89e-b843fb19b2e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:12.957428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c21269e6-0190-4904-a89e-b843fb19b2e4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:12.995832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:13.046745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21269e6-0190-4904-a89e-b843fb19b2e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:13.048385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21269e6-0190-4904-a89e-b843fb19b2e4 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:13.067458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 697/1388] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:13.617214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b23102-402c-4493-8e91-9143aa665554 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:13.617877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b23102-402c-4493-8e91-9143aa665554 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:13.617877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b23102-402c-4493-8e91-9143aa665554 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:13.619064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 692/1389] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:13.624921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c70004-ee09-4c86-9804-3994adad6df2 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:13.625366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c70004-ee09-4c86-9804-3994adad6df2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:13.625468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c70004-ee09-4c86-9804-3994adad6df2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:13.692739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-60c70004-ee09-4c86-9804-3994adad6df2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:13.693940 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-60c70004-ee09-4c86-9804-3994adad6df2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:13.796621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 698/1390] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:14.099613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a162bf1-1546-4d36-9bc3-4fcc2b9a2d5e None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:14.099898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a162bf1-1546-4d36-9bc3-4fcc2b9a2d5e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:14.100203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a162bf1-1546-4d36-9bc3-4fcc2b9a2d5e None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:14.101532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 693/1391] 10.222.0.42 () {54 vars in 962 bytes} [Fri Sep 16 01:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:14.106890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa7ac9-0b04-482e-ba5b-b66fb58f0520 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:14.107389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa7ac9-0b04-482e-ba5b-b66fb58f0520 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:14.107669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aa7ac9-0b04-482e-ba5b-b66fb58f0520 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:14.124069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:14.184343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a7aa7ac9-0b04-482e-ba5b-b66fb58f0520 None None] MFA Rules not processed for user `30b10ebdc87a4747bef7fc60b3cec9c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:14.186697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a7aa7ac9-0b04-482e-ba5b-b66fb58f0520 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:14.261664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 699/1392] 10.222.0.42 () {58 vars in 1059 bytes} [Fri Sep 16 01:47:14 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:14.270618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a162bf1-1546-4d36-9bc3-4fcc2b9a2d5e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:14.272172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4a162bf1-1546-4d36-9bc3-4fcc2b9a2d5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:14.376735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a162bf1-1546-4d36-9bc3-4fcc2b9a2d5e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:14.435658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:14.452896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa95e51-2c15-4ded-9aba-96e9eac023b6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:14.453488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa95e51-2c15-4ded-9aba-96e9eac023b6 service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:14.453761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa95e51-2c15-4ded-9aba-96e9eac023b6 service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:14.454134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa95e51-2c15-4ded-9aba-96e9eac023b6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:14.455698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1aa95e51-2c15-4ded-9aba-96e9eac023b6 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:14.565373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa95e51-2c15-4ded-9aba-96e9eac023b6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:14.566824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa95e51-2c15-4ded-9aba-96e9eac023b6 service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:14.585449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 694/1393] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:15.315210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3295183-7ff3-4db5-add9-36ed07c9a578 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:15.315791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3295183-7ff3-4db5-add9-36ed07c9a578 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:15.315791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3295183-7ff3-4db5-add9-36ed07c9a578 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:15.317232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 700/1394] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:15.322349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b2a841-3d0d-4830-b450-92f0748f3b0c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:15.322768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b2a841-3d0d-4830-b450-92f0748f3b0c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:15.322880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b2a841-3d0d-4830-b450-92f0748f3b0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:15.340603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdd792f-ad4c-4157-a554-4b863fa2c717 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:15.341439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdd792f-ad4c-4157-a554-4b863fa2c717 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:15.341753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdd792f-ad4c-4157-a554-4b863fa2c717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:15.393065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b2b2a841-3d0d-4830-b450-92f0748f3b0c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:15.394267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b2b2a841-3d0d-4830-b450-92f0748f3b0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:15.411838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-fcdd792f-ad4c-4157-a554-4b863fa2c717 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:15.413063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fcdd792f-ad4c-4157-a554-4b863fa2c717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:15.438766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:15.471334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:15.497002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 695/1395] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:15.504304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 701/1396] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:47:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:15.547073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb8201-37d7-4198-b5a9-272e46084911 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:15.547355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb8201-37d7-4198-b5a9-272e46084911 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:15.547627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbb8201-37d7-4198-b5a9-272e46084911 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:15.549032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 696/1397] 10.222.0.42 () {54 vars in 943 bytes} [Fri Sep 16 01:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:15.556170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:15.556465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:15.556739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:15.634091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] MFA Rules not processed for user `af7a48729497436784b988c6291cfa5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:15.635306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:15.741487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 702/1398] 10.222.0.42 () {58 vars in 1040 bytes} [Fri Sep 16 01:47:15 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:47:15.750923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb8201-37d7-4198-b5a9-272e46084911 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:15.752887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-afbb8201-37d7-4198-b5a9-272e46084911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:15.798108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:15.799290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:15.832590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:15.894486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbb8201-37d7-4198-b5a9-272e46084911 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:15.901332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52740eed-6811-4081-b278-33f1863a57f7 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:15.938999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:16.007365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c277efe-9de4-46f2-87ef-e17226310627 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:16.007980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c277efe-9de4-46f2-87ef-e17226310627 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:16.008262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c277efe-9de4-46f2-87ef-e17226310627 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:16.008789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c277efe-9de4-46f2-87ef-e17226310627 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:16.010405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9c277efe-9de4-46f2-87ef-e17226310627 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:16.022156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0879f180-3aa0-4b8e-839d-564d8f93f434 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:16.022760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879f180-3aa0-4b8e-839d-564d8f93f434 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:16.023038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879f180-3aa0-4b8e-839d-564d8f93f434 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:16.023309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0879f180-3aa0-4b8e-839d-564d8f93f434 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:16.024934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0879f180-3aa0-4b8e-839d-564d8f93f434 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:16.069210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:16.098063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c277efe-9de4-46f2-87ef-e17226310627 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:16.099664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c277efe-9de4-46f2-87ef-e17226310627 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:16.108531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 697/1399] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:47:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:47:16.118687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0879f180-3aa0-4b8e-839d-564d8f93f434 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:16.120325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0879f180-3aa0-4b8e-839d-564d8f93f434 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:16.139541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 703/1400] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:47:18.379962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e2f5b5-6e18-4d0d-ad88-1fd3bb8bc60f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:18.380524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e2f5b5-6e18-4d0d-ad88-1fd3bb8bc60f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:18.380524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e2f5b5-6e18-4d0d-ad88-1fd3bb8bc60f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:18.381909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 698/1401] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:18.391814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e231f49-e45c-440e-8ced-ec967f975332 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:18.393178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e231f49-e45c-440e-8ced-ec967f975332 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:18.393464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e231f49-e45c-440e-8ced-ec967f975332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:18.491042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-9e231f49-e45c-440e-8ced-ec967f975332 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:18.492542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9e231f49-e45c-440e-8ced-ec967f975332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:18.616087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 704/1402] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:18.905384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e2f5b5-6e18-4d0d-ad88-1fd3bb8bc60f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:18.906896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e9e2f5b5-6e18-4d0d-ad88-1fd3bb8bc60f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:18.991337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:19.072068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e2f5b5-6e18-4d0d-ad88-1fd3bb8bc60f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:19.214541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6129efd-0059-4727-aab9-92f234ce2967 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:19.215221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6129efd-0059-4727-aab9-92f234ce2967 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:19.215503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6129efd-0059-4727-aab9-92f234ce2967 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:19.215798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6129efd-0059-4727-aab9-92f234ce2967 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:19.217740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d6129efd-0059-4727-aab9-92f234ce2967 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:19.256234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:19.324740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6129efd-0059-4727-aab9-92f234ce2967 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:19.326378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6129efd-0059-4727-aab9-92f234ce2967 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:19.346091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 699/1403] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:21.615038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5c9186-c498-4401-b917-30eb1fb8413c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:21.615599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5c9186-c498-4401-b917-30eb1fb8413c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:21.615599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5c9186-c498-4401-b917-30eb1fb8413c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:21.619985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 705/1404] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:21.623078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82373fb0-9c10-4aa2-8625-a9e325e61990 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:21.623510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82373fb0-9c10-4aa2-8625-a9e325e61990 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:21.623615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82373fb0-9c10-4aa2-8625-a9e325e61990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:21.696294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-82373fb0-9c10-4aa2-8625-a9e325e61990 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:21.697409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-82373fb0-9c10-4aa2-8625-a9e325e61990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:21.815458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 700/1405] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:22.109169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5c9186-c498-4401-b917-30eb1fb8413c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:22.110679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4d5c9186-c498-4401-b917-30eb1fb8413c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:22.135096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:22.280205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d5c9186-c498-4401-b917-30eb1fb8413c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:22.390313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:22.423167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfca0b50-4390-45ba-8f6f-80b31217b556 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:22.424248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfca0b50-4390-45ba-8f6f-80b31217b556 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:22.424738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfca0b50-4390-45ba-8f6f-80b31217b556 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:22.425226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfca0b50-4390-45ba-8f6f-80b31217b556 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:22.427323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bfca0b50-4390-45ba-8f6f-80b31217b556 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:22.530760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfca0b50-4390-45ba-8f6f-80b31217b556 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:22.532385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfca0b50-4390-45ba-8f6f-80b31217b556 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:22.552171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 706/1406] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:24.839788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc4c40-7947-4afe-92ed-604381d79dcb None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:24.840699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc4c40-7947-4afe-92ed-604381d79dcb None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:24.840975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc4c40-7947-4afe-92ed-604381d79dcb None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:24.842303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 701/1407] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:47:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:47:24.847854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1405b-1c1d-413b-928c-f178186bc843 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:24.848337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1405b-1c1d-413b-928c-f178186bc843 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:24.848432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f1405b-1c1d-413b-928c-f178186bc843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:24.921767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-61f1405b-1c1d-413b-928c-f178186bc843 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:24.923831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-61f1405b-1c1d-413b-928c-f178186bc843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:24.986222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:25.032445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 707/1408] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:47:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:25.333368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc4c40-7947-4afe-92ed-604381d79dcb None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:25.334893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8adc4c40-7947-4afe-92ed-604381d79dcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:25.343163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:25.516188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc4c40-7947-4afe-92ed-604381d79dcb None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:25.610924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:25.648605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74931e3f-ca4b-4dbf-b41f-804e67ea9a17 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:25.649227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74931e3f-ca4b-4dbf-b41f-804e67ea9a17 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:25.649517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74931e3f-ca4b-4dbf-b41f-804e67ea9a17 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:25.649887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74931e3f-ca4b-4dbf-b41f-804e67ea9a17 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:25.651474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-74931e3f-ca4b-4dbf-b41f-804e67ea9a17 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:25.747712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74931e3f-ca4b-4dbf-b41f-804e67ea9a17 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:25.749363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74931e3f-ca4b-4dbf-b41f-804e67ea9a17 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:25.772344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 702/1409] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:35.030846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=19 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:35.765939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=59 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:39.161343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dea69e-7fba-4031-81d7-7eba975db6bb None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:39.161845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dea69e-7fba-4031-81d7-7eba975db6bb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:39.161960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dea69e-7fba-4031-81d7-7eba975db6bb None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:39.163242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 708/1410] 10.222.0.42 () {54 vars in 918 bytes} [Fri Sep 16 01:47:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:39.168254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def1b768-4228-4260-886a-44280863b095 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:39.168687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def1b768-4228-4260-886a-44280863b095 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:39.168798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def1b768-4228-4260-886a-44280863b095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:39.245522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-def1b768-4228-4260-886a-44280863b095 None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:39.247877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-def1b768-4228-4260-886a-44280863b095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:39.354041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 703/1411] 10.222.0.42 () {58 vars in 1015 bytes} [Fri Sep 16 01:47:39 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:39.368810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae2c00-b619-41a4-b04e-700764384650 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:39.369102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae2c00-b619-41a4-b04e-700764384650 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:39.369379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ae2c00-b619-41a4-b04e-700764384650 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:39.370697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 709/1412] 10.222.0.42 () {54 vars in 918 bytes} [Fri Sep 16 01:47:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:39.375366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b254d3d5-0321-4333-b5dd-d054281cefe2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:39.375651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b254d3d5-0321-4333-b5dd-d054281cefe2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:39.375922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b254d3d5-0321-4333-b5dd-d054281cefe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:39.450568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:39.453656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b254d3d5-0321-4333-b5dd-d054281cefe2 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:39.455447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b254d3d5-0321-4333-b5dd-d054281cefe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:39.569908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 704/1413] 10.222.0.42 () {58 vars in 1015 bytes} [Fri Sep 16 01:47:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:42.142551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546adb5b-d645-4e23-9d7a-f79dc8925ee7 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:42.143098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546adb5b-d645-4e23-9d7a-f79dc8925ee7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:42.143098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546adb5b-d645-4e23-9d7a-f79dc8925ee7 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:42.144514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 710/1414] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:42.149781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e2c59-2a98-4396-b56b-15e1bf093f5b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:42.150223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e2c59-2a98-4396-b56b-15e1bf093f5b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:42.150352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1e2c59-2a98-4396-b56b-15e1bf093f5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:42.216242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ff1e2c59-2a98-4396-b56b-15e1bf093f5b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:42.217436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ff1e2c59-2a98-4396-b56b-15e1bf093f5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:42.271832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:42.313613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 705/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:47:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:42.612970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546adb5b-d645-4e23-9d7a-f79dc8925ee7 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:42.614573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-546adb5b-d645-4e23-9d7a-f79dc8925ee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:42.762088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546adb5b-d645-4e23-9d7a-f79dc8925ee7 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:42.836319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:42.879073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52583855-d982-4519-a453-ddad2ed1c4dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:42.879666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52583855-d982-4519-a453-ddad2ed1c4dc service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:42.879969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52583855-d982-4519-a453-ddad2ed1c4dc service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:42.880227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52583855-d982-4519-a453-ddad2ed1c4dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:42.881789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-52583855-d982-4519-a453-ddad2ed1c4dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:42.969467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52583855-d982-4519-a453-ddad2ed1c4dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:42.971060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52583855-d982-4519-a453-ddad2ed1c4dc service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:42.989079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 711/1416] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:44.129618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0b24e5-b601-4d73-a68a-bad7b3cff379 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:44.130139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0b24e5-b601-4d73-a68a-bad7b3cff379 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:44.130254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0b24e5-b601-4d73-a68a-bad7b3cff379 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:44.131531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 706/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:44.138757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63668831-97fb-40d0-b3bc-38b116f896e9 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:44.139163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63668831-97fb-40d0-b3bc-38b116f896e9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:44.139293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63668831-97fb-40d0-b3bc-38b116f896e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:44.214191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-63668831-97fb-40d0-b3bc-38b116f896e9 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:44.215985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-63668831-97fb-40d0-b3bc-38b116f896e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:44.242080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:44.311515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 712/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:44.462448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0b24e5-b601-4d73-a68a-bad7b3cff379 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:44.464065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-da0b24e5-b601-4d73-a68a-bad7b3cff379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:44.609746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0b24e5-b601-4d73-a68a-bad7b3cff379 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:44.650099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:44.725676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57067d44-6abd-4c09-b706-c861663042a4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:44.726188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57067d44-6abd-4c09-b706-c861663042a4 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:44.726457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57067d44-6abd-4c09-b706-c861663042a4 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:44.726720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57067d44-6abd-4c09-b706-c861663042a4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:44.728320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-57067d44-6abd-4c09-b706-c861663042a4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:44.819381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57067d44-6abd-4c09-b706-c861663042a4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:44.821029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57067d44-6abd-4c09-b706-c861663042a4 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:44.838995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 707/1419] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:45.967276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1785a927-ec49-4e38-9d4d-b3f3f54374a5 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:45.967760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1785a927-ec49-4e38-9d4d-b3f3f54374a5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:45.967872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1785a927-ec49-4e38-9d4d-b3f3f54374a5 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:45.969188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 713/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:45.975557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb6a52-3549-45d7-a141-2374c6e666e5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:45.975979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb6a52-3549-45d7-a141-2374c6e666e5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:45.976133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fb6a52-3549-45d7-a141-2374c6e666e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:46.030983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:46.052890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-92fb6a52-3549-45d7-a141-2374c6e666e5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:46.054083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-92fb6a52-3549-45d7-a141-2374c6e666e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:46.161851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 708/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:46.313798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1785a927-ec49-4e38-9d4d-b3f3f54374a5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:46.315303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1785a927-ec49-4e38-9d4d-b3f3f54374a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:46.460023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1785a927-ec49-4e38-9d4d-b3f3f54374a5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:46.472044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:46.574678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0017d26-f384-46ef-b995-ebe7747c90f8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:46.575270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0017d26-f384-46ef-b995-ebe7747c90f8 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:46.575541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0017d26-f384-46ef-b995-ebe7747c90f8 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:46.575806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0017d26-f384-46ef-b995-ebe7747c90f8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:46.577398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e0017d26-f384-46ef-b995-ebe7747c90f8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:46.666333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0017d26-f384-46ef-b995-ebe7747c90f8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:46.668042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0017d26-f384-46ef-b995-ebe7747c90f8 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:46.685864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 714/1422] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:47.795146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fd86fa-7d38-4938-9e33-3617aac0b6b6 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:47.795733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fd86fa-7d38-4938-9e33-3617aac0b6b6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:47.795733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fd86fa-7d38-4938-9e33-3617aac0b6b6 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:47.799747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 709/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:47.803848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270f881-cdc8-407b-ae22-05f0d7d0510a None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:47.804305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270f881-cdc8-407b-ae22-05f0d7d0510a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:47.804410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e270f881-cdc8-407b-ae22-05f0d7d0510a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:47.824591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:47.880064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e270f881-cdc8-407b-ae22-05f0d7d0510a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:47.881266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e270f881-cdc8-407b-ae22-05f0d7d0510a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:47.976804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 715/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:48.125397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fd86fa-7d38-4938-9e33-3617aac0b6b6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:48.126908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c3fd86fa-7d38-4938-9e33-3617aac0b6b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:48.237685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:48.273957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fd86fa-7d38-4938-9e33-3617aac0b6b6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:48.386420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c961638-4fae-4e0f-83e5-0059c124cd9a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:48.387496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c961638-4fae-4e0f-83e5-0059c124cd9a service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:48.387777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c961638-4fae-4e0f-83e5-0059c124cd9a service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:48.388067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c961638-4fae-4e0f-83e5-0059c124cd9a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:48.389669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0c961638-4fae-4e0f-83e5-0059c124cd9a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:48.466793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:48.480544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c961638-4fae-4e0f-83e5-0059c124cd9a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:48.482152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c961638-4fae-4e0f-83e5-0059c124cd9a service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:48.500236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 710/1425] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:49.626528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04caede2-6b98-4f76-8896-b79826302592 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:49.627131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04caede2-6b98-4f76-8896-b79826302592 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:49.627131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04caede2-6b98-4f76-8896-b79826302592 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:49.628444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 716/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:49.634893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee73cd98-aea9-4625-aa50-40001522375c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:49.635326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee73cd98-aea9-4625-aa50-40001522375c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:49.635438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee73cd98-aea9-4625-aa50-40001522375c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:49.711666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ee73cd98-aea9-4625-aa50-40001522375c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:49.712818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ee73cd98-aea9-4625-aa50-40001522375c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:49.811879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 711/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:49.962699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04caede2-6b98-4f76-8896-b79826302592 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:49.964156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-04caede2-6b98-4f76-8896-b79826302592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:50.049044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:50.110954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04caede2-6b98-4f76-8896-b79826302592 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:50.221634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf73335-cd16-4746-8ddd-ece6a4c12015 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:50.222147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf73335-cd16-4746-8ddd-ece6a4c12015 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:50.222494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf73335-cd16-4746-8ddd-ece6a4c12015 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:50.222684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf73335-cd16-4746-8ddd-ece6a4c12015 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:50.224269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4bf73335-cd16-4746-8ddd-ece6a4c12015 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:50.271755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:50.312927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf73335-cd16-4746-8ddd-ece6a4c12015 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:50.314505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf73335-cd16-4746-8ddd-ece6a4c12015 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:50.332804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 717/1428] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:51.394710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f4f3f-130b-46b3-9001-2cd07f71c05d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:51.395261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f4f3f-130b-46b3-9001-2cd07f71c05d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:51.395261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1f4f3f-130b-46b3-9001-2cd07f71c05d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:51.396594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 712/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:47:51.403339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35756d94-4a4a-40f8-a7ed-93192782cded None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:51.403786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35756d94-4a4a-40f8-a7ed-93192782cded None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:51.403894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35756d94-4a4a-40f8-a7ed-93192782cded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:51.484688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-35756d94-4a4a-40f8-a7ed-93192782cded None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:51.485908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-35756d94-4a4a-40f8-a7ed-93192782cded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:51.582765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 718/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:51.735602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1f4f3f-130b-46b3-9001-2cd07f71c05d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:51.737064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eb1f4f3f-130b-46b3-9001-2cd07f71c05d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:51.788840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:51.890163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1f4f3f-130b-46b3-9001-2cd07f71c05d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:52.010632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8c13d8-894e-4c45-b76e-1cf35c984234 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:52.011699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c13d8-894e-4c45-b76e-1cf35c984234 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:52.012001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c13d8-894e-4c45-b76e-1cf35c984234 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:52.012278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c13d8-894e-4c45-b76e-1cf35c984234 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:52.014045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9b8c13d8-894e-4c45-b76e-1cf35c984234 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:52.021258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:52.104807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8c13d8-894e-4c45-b76e-1cf35c984234 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:52.106509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8c13d8-894e-4c45-b76e-1cf35c984234 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:52.124554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 713/1431] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:53.180564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91634265-479c-4a11-b70c-861b7a58e354 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:53.181099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91634265-479c-4a11-b70c-861b7a58e354 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:53.181099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91634265-479c-4a11-b70c-861b7a58e354 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:53.182443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 719/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:53.189233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482a94a4-b84b-4d5d-92e5-e6b6a223ca77 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:53.189660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482a94a4-b84b-4d5d-92e5-e6b6a223ca77 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:53.189764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482a94a4-b84b-4d5d-92e5-e6b6a223ca77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:53.266068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-482a94a4-b84b-4d5d-92e5-e6b6a223ca77 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:53.267274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-482a94a4-b84b-4d5d-92e5-e6b6a223ca77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:53.337075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:53.366717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 714/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:53.520124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91634265-479c-4a11-b70c-861b7a58e354 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:53.521572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-91634265-479c-4a11-b70c-861b7a58e354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:53.539346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:53.666621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91634265-479c-4a11-b70c-861b7a58e354 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:53.753146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:53.779751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38f13bd-4d8f-4cb3-92c4-e382a7e00ee3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:53.780314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f13bd-4d8f-4cb3-92c4-e382a7e00ee3 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:53.780589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f13bd-4d8f-4cb3-92c4-e382a7e00ee3 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:53.780854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38f13bd-4d8f-4cb3-92c4-e382a7e00ee3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:53.782435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e38f13bd-4d8f-4cb3-92c4-e382a7e00ee3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:53.871904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f13bd-4d8f-4cb3-92c4-e382a7e00ee3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:53.873543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38f13bd-4d8f-4cb3-92c4-e382a7e00ee3 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:53.892059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 720/1434] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:54.952675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc232be-b53a-418b-9d8c-9700b3a0dd04 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:54.953241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc232be-b53a-418b-9d8c-9700b3a0dd04 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:54.953241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc232be-b53a-418b-9d8c-9700b3a0dd04 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:54.954604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 715/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:54.960990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e9053e-5fab-46a2-9e14-33e570f8aacc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:54.961436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e9053e-5fab-46a2-9e14-33e570f8aacc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:54.961534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e9053e-5fab-46a2-9e14-33e570f8aacc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:55.037413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-05e9053e-5fab-46a2-9e14-33e570f8aacc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:55.038608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-05e9053e-5fab-46a2-9e14-33e570f8aacc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:55.080065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:55.133550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 721/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:55.282788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc232be-b53a-418b-9d8c-9700b3a0dd04 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:55.284249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ebc232be-b53a-418b-9d8c-9700b3a0dd04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:55.428212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc232be-b53a-418b-9d8c-9700b3a0dd04 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:55.491483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:55.550262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fcaed1b-0166-4196-9499-439d06244d0d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:55.550784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcaed1b-0166-4196-9499-439d06244d0d service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:55.551058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcaed1b-0166-4196-9499-439d06244d0d service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:55.551321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fcaed1b-0166-4196-9499-439d06244d0d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:55.552938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0fcaed1b-0166-4196-9499-439d06244d0d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:55.641935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcaed1b-0166-4196-9499-439d06244d0d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:55.643516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fcaed1b-0166-4196-9499-439d06244d0d service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:55.660894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 716/1437] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:56.732572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2c1f3-6a61-44db-9b87-3e496ec9e73a None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:56.733164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2c1f3-6a61-44db-9b87-3e496ec9e73a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:56.733164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db2c1f3-6a61-44db-9b87-3e496ec9e73a None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:56.734431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 722/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:47:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:56.740666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c715c5b5-c22d-4afb-9a4b-220b874a668d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:56.741105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c715c5b5-c22d-4afb-9a4b-220b874a668d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:56.741209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c715c5b5-c22d-4afb-9a4b-220b874a668d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:56.818981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c715c5b5-c22d-4afb-9a4b-220b874a668d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:56.820197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c715c5b5-c22d-4afb-9a4b-220b874a668d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:56.822150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:56.915731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 717/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:47:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:57.064594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db2c1f3-6a61-44db-9b87-3e496ec9e73a None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:57.066028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4db2c1f3-6a61-44db-9b87-3e496ec9e73a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:57.210934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db2c1f3-6a61-44db-9b87-3e496ec9e73a None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:47:57.237936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:57.322788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b166492-88e4-420c-b89a-f3b93663a93d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:47:57.323305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b166492-88e4-420c-b89a-f3b93663a93d service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:57.323627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b166492-88e4-420c-b89a-f3b93663a93d service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:57.323898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b166492-88e4-420c-b89a-f3b93663a93d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:57.325584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4b166492-88e4-420c-b89a-f3b93663a93d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:57.414406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b166492-88e4-420c-b89a-f3b93663a93d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:47:57.416016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b166492-88e4-420c-b89a-f3b93663a93d service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:47:57.434810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 723/1440] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:47:59.480684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3df329-1ec0-4eb7-b6bb-8fab950e9148 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:59.481246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3df329-1ec0-4eb7-b6bb-8fab950e9148 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:59.481246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3df329-1ec0-4eb7-b6bb-8fab950e9148 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:59.482440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 718/1441] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:47:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:47:59.488291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ec3087-abf7-44f8-bfe1-084f30955af2 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:47:59.488729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ec3087-abf7-44f8-bfe1-084f30955af2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:47:59.488856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ec3087-abf7-44f8-bfe1-084f30955af2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:47:59.530462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:47:59.555927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-78ec3087-abf7-44f8-bfe1-084f30955af2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:47:59.557160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-78ec3087-abf7-44f8-bfe1-084f30955af2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:47:59.653376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 724/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:47:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:47:59.940530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3df329-1ec0-4eb7-b6bb-8fab950e9148 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:47:59.942090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2b3df329-1ec0-4eb7-b6bb-8fab950e9148 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:00.044958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3df329-1ec0-4eb7-b6bb-8fab950e9148 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:00.077450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:00.118128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dec480-3ff8-48e3-b791-5da62651e5e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:00.118718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dec480-3ff8-48e3-b791-5da62651e5e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:00.118989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dec480-3ff8-48e3-b791-5da62651e5e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:00.119251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dec480-3ff8-48e3-b791-5da62651e5e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:00.120879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-11dec480-3ff8-48e3-b791-5da62651e5e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:00.227992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dec480-3ff8-48e3-b791-5da62651e5e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:00.229421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dec480-3ff8-48e3-b791-5da62651e5e0 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:00.247172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 719/1443] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:47:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:02.555563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecec52a-0688-497f-a987-aa837a7718ee None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:02.556158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecec52a-0688-497f-a987-aa837a7718ee None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:02.556158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecec52a-0688-497f-a987-aa837a7718ee None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:02.557470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 725/1444] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:02.563455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703c6a04-66be-4da9-a563-fad76427ade9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:02.563846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703c6a04-66be-4da9-a563-fad76427ade9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:02.564077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703c6a04-66be-4da9-a563-fad76427ade9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:02.629585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-703c6a04-66be-4da9-a563-fad76427ade9 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:02.630791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-703c6a04-66be-4da9-a563-fad76427ade9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:02.644332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:02.730775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 720/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:03.016007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecec52a-0688-497f-a987-aa837a7718ee None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:03.017554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-eecec52a-0688-497f-a987-aa837a7718ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:03.120623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecec52a-0688-497f-a987-aa837a7718ee None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:03.133255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:03.194892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d3a8f-00fc-41f8-8edc-de2a229454c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:03.195544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d3a8f-00fc-41f8-8edc-de2a229454c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:03.195849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d3a8f-00fc-41f8-8edc-de2a229454c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:03.196151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d3a8f-00fc-41f8-8edc-de2a229454c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:03.197753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fa8d3a8f-00fc-41f8-8edc-de2a229454c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:03.305036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8d3a8f-00fc-41f8-8edc-de2a229454c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:03.306453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8d3a8f-00fc-41f8-8edc-de2a229454c3 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:03.325182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 726/1446] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:05.794970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f461d-5a77-4799-b8ed-cdfc7a792459 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:05.795556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f461d-5a77-4799-b8ed-cdfc7a792459 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:05.795556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3f461d-5a77-4799-b8ed-cdfc7a792459 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:05.797021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 721/1447] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:05.802385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffab02d-63c8-4249-91e0-bda73667edd1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:05.802847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffab02d-63c8-4249-91e0-bda73667edd1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:05.802956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffab02d-63c8-4249-91e0-bda73667edd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:05.852149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:05.871117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0ffab02d-63c8-4249-91e0-bda73667edd1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:05.872358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0ffab02d-63c8-4249-91e0-bda73667edd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:05.968479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 727/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:48:06.257243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f461d-5a77-4799-b8ed-cdfc7a792459 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:06.259019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ae3f461d-5a77-4799-b8ed-cdfc7a792459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:06.361867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3f461d-5a77-4799-b8ed-cdfc7a792459 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:06.403389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:06.438334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede7b3e9-7401-4a1e-8dd7-47855797c1a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:06.438931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7b3e9-7401-4a1e-8dd7-47855797c1a8 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:06.439205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7b3e9-7401-4a1e-8dd7-47855797c1a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:06.439469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede7b3e9-7401-4a1e-8dd7-47855797c1a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:06.441094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ede7b3e9-7401-4a1e-8dd7-47855797c1a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:06.549783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede7b3e9-7401-4a1e-8dd7-47855797c1a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:06.551123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede7b3e9-7401-4a1e-8dd7-47855797c1a8 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:06.569244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 722/1449] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:08.936917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635d098-8f93-46b8-b786-0bd394360e0f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:08.937445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635d098-8f93-46b8-b786-0bd394360e0f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:08.937541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f635d098-8f93-46b8-b786-0bd394360e0f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:08.938827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 728/1450] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:08.944237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37408a0-f62f-4c57-920c-1252935bf6e6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:08.944643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37408a0-f62f-4c57-920c-1252935bf6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:08.944780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37408a0-f62f-4c57-920c-1252935bf6e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:09.010102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d37408a0-f62f-4c57-920c-1252935bf6e6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:09.012083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d37408a0-f62f-4c57-920c-1252935bf6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:09.032898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:09.109278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 723/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:09.395719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f635d098-8f93-46b8-b786-0bd394360e0f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:09.397303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f635d098-8f93-46b8-b786-0bd394360e0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:09.501934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f635d098-8f93-46b8-b786-0bd394360e0f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:09.522709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:09.576640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b762ff65-60be-4b31-86f3-c910f2b0f3c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:09.577234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b762ff65-60be-4b31-86f3-c910f2b0f3c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:09.577507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b762ff65-60be-4b31-86f3-c910f2b0f3c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:09.577771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b762ff65-60be-4b31-86f3-c910f2b0f3c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:09.579399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b762ff65-60be-4b31-86f3-c910f2b0f3c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:09.687971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b762ff65-60be-4b31-86f3-c910f2b0f3c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:09.689331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b762ff65-60be-4b31-86f3-c910f2b0f3c7 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:09.707411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 729/1452] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:11.955217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be42a8d4-39c0-4ef4-a07c-46280788559c None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:11.955773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be42a8d4-39c0-4ef4-a07c-46280788559c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:11.955773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be42a8d4-39c0-4ef4-a07c-46280788559c None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:11.957089 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 724/1453] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:11.962368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc96e6-4be8-420c-b9e9-3ceda6f885ab None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:11.962794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc96e6-4be8-420c-b9e9-3ceda6f885ab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:11.962906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc96e6-4be8-420c-b9e9-3ceda6f885ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:12.027203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:12.030034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-63fc96e6-4be8-420c-b9e9-3ceda6f885ab None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:12.031764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-63fc96e6-4be8-420c-b9e9-3ceda6f885ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:12.127098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 730/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:48:12.419925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be42a8d4-39c0-4ef4-a07c-46280788559c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:12.421481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-be42a8d4-39c0-4ef4-a07c-46280788559c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:12.526029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be42a8d4-39c0-4ef4-a07c-46280788559c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:12.575626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:12.599513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d1f125-d69a-4726-9501-b5a3ceb6cae7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:12.600133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d1f125-d69a-4726-9501-b5a3ceb6cae7 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:12.600411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d1f125-d69a-4726-9501-b5a3ceb6cae7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:12.600674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d1f125-d69a-4726-9501-b5a3ceb6cae7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:12.602251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f2d1f125-d69a-4726-9501-b5a3ceb6cae7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:12.709711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d1f125-d69a-4726-9501-b5a3ceb6cae7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:12.711088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d1f125-d69a-4726-9501-b5a3ceb6cae7 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:12.728713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 725/1455] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:14.927952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbed30af-a8fb-4a58-b2b5-3a3c355272b3 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:14.928510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbed30af-a8fb-4a58-b2b5-3a3c355272b3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:14.928510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbed30af-a8fb-4a58-b2b5-3a3c355272b3 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:14.929909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 731/1456] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:14.935841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd5c76e-3611-4edf-87d6-a9d568038a9d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:14.936272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd5c76e-3611-4edf-87d6-a9d568038a9d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:14.936423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd5c76e-3611-4edf-87d6-a9d568038a9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:15.002654 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0bd5c76e-3611-4edf-87d6-a9d568038a9d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:15.003845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0bd5c76e-3611-4edf-87d6-a9d568038a9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:15.033435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:15.100643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 726/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:15.389901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbed30af-a8fb-4a58-b2b5-3a3c355272b3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:15.391448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-dbed30af-a8fb-4a58-b2b5-3a3c355272b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:15.493555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbed30af-a8fb-4a58-b2b5-3a3c355272b3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:15.524880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:15.686656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b7b726-9305-4138-8997-35e5e8f13997 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:15.687155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7b726-9305-4138-8997-35e5e8f13997 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:15.687430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7b726-9305-4138-8997-35e5e8f13997 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:15.687693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b7b726-9305-4138-8997-35e5e8f13997 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:15.689299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-82b7b726-9305-4138-8997-35e5e8f13997 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:15.798290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b7b726-9305-4138-8997-35e5e8f13997 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:15.799718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b7b726-9305-4138-8997-35e5e8f13997 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:15.818095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 732/1458] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:18.068033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8c8fc-f3b2-43f3-9404-235cebf49636 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:18.068567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8c8fc-f3b2-43f3-9404-235cebf49636 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:18.068567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8c8fc-f3b2-43f3-9404-235cebf49636 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:18.069871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 727/1459] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:18.075830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08fe63f-8c2a-4ecb-b5ba-6439091c615b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:18.076308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08fe63f-8c2a-4ecb-b5ba-6439091c615b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:18.076404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08fe63f-8c2a-4ecb-b5ba-6439091c615b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:18.141261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c08fe63f-8c2a-4ecb-b5ba-6439091c615b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:18.142456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c08fe63f-8c2a-4ecb-b5ba-6439091c615b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:18.150500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:18.237812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 733/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:18.525203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8c8fc-f3b2-43f3-9404-235cebf49636 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:18.527288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2cf8c8fc-f3b2-43f3-9404-235cebf49636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:18.632233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8c8fc-f3b2-43f3-9404-235cebf49636 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:18.688857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:18.707044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7999bbd2-9448-4b62-b56d-e78b1769e05f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:18.707635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7999bbd2-9448-4b62-b56d-e78b1769e05f service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:18.707908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7999bbd2-9448-4b62-b56d-e78b1769e05f service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:18.708201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7999bbd2-9448-4b62-b56d-e78b1769e05f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:18.709766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7999bbd2-9448-4b62-b56d-e78b1769e05f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:18.821062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7999bbd2-9448-4b62-b56d-e78b1769e05f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:18.822579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7999bbd2-9448-4b62-b56d-e78b1769e05f service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:18.840741 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 728/1461] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:21.052793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5ffb1f-e5a8-4438-95fa-f797dd302678 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:21.053326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5ffb1f-e5a8-4438-95fa-f797dd302678 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:21.053326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5ffb1f-e5a8-4438-95fa-f797dd302678 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:21.054707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 734/1462] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:48:21.060371 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608fa2f-5a4f-4e99-b5dd-63930abbdee3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:21.060826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608fa2f-5a4f-4e99-b5dd-63930abbdee3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:21.060937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2608fa2f-5a4f-4e99-b5dd-63930abbdee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:21.126788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2608fa2f-5a4f-4e99-b5dd-63930abbdee3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:21.127994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2608fa2f-5a4f-4e99-b5dd-63930abbdee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:21.167593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:21.225196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 729/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:48:21.513135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5ffb1f-e5a8-4438-95fa-f797dd302678 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:21.514658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fb5ffb1f-e5a8-4438-95fa-f797dd302678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:21.625447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5ffb1f-e5a8-4438-95fa-f797dd302678 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:21.663870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:21.699520 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f263f0dc-261f-45cf-94dd-6687b205e586 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:21.700146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f263f0dc-261f-45cf-94dd-6687b205e586 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:21.700426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f263f0dc-261f-45cf-94dd-6687b205e586 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:21.700689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f263f0dc-261f-45cf-94dd-6687b205e586 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:21.702251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f263f0dc-261f-45cf-94dd-6687b205e586 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:21.810508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f263f0dc-261f-45cf-94dd-6687b205e586 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:21.811974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f263f0dc-261f-45cf-94dd-6687b205e586 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:21.829823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 735/1464] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:23.986511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3bf59b-d1e0-4648-9fb3-921e80f51a05 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:23.987070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3bf59b-d1e0-4648-9fb3-921e80f51a05 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:23.987175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3bf59b-d1e0-4648-9fb3-921e80f51a05 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:23.988637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 730/1465] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:23.993925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4787da43-ac5e-4040-b336-3c797256bec8 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:23.995587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4787da43-ac5e-4040-b336-3c797256bec8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:23.995587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4787da43-ac5e-4040-b336-3c797256bec8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:24.059982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4787da43-ac5e-4040-b336-3c797256bec8 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:24.061094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4787da43-ac5e-4040-b336-3c797256bec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:24.078376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:24.156074 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 736/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:24.437500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3bf59b-d1e0-4648-9fb3-921e80f51a05 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:24.439033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0c3bf59b-d1e0-4648-9fb3-921e80f51a05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:24.542648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3bf59b-d1e0-4648-9fb3-921e80f51a05 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:24.610219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:24.617225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c5208a-5c4a-4502-a183-dc09acf15c73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:24.617820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c5208a-5c4a-4502-a183-dc09acf15c73 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:24.618093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c5208a-5c4a-4502-a183-dc09acf15c73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:24.618354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c5208a-5c4a-4502-a183-dc09acf15c73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:24.620487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-65c5208a-5c4a-4502-a183-dc09acf15c73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:24.729270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c5208a-5c4a-4502-a183-dc09acf15c73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:24.730821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c5208a-5c4a-4502-a183-dc09acf15c73 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:24.749019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 731/1467] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:26.938999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1300f00f-090a-42f8-bca2-1e59fd2563fb None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:26.939553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1300f00f-090a-42f8-bca2-1e59fd2563fb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:26.939553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1300f00f-090a-42f8-bca2-1e59fd2563fb None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:26.940923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 737/1468] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:48:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:26.946171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1811a4da-c53a-431e-9028-f22b2300a41e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:26.946570 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1811a4da-c53a-431e-9028-f22b2300a41e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:26.946713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1811a4da-c53a-431e-9028-f22b2300a41e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:27.013817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1811a4da-c53a-431e-9028-f22b2300a41e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:27.015028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1811a4da-c53a-431e-9028-f22b2300a41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:27.063586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:27.111755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 732/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:48:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:27.397027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1300f00f-090a-42f8-bca2-1e59fd2563fb None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:27.398554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1300f00f-090a-42f8-bca2-1e59fd2563fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:27.500968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1300f00f-090a-42f8-bca2-1e59fd2563fb None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:27.549349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:27.575897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59915bff-b05f-4cd9-94fd-08c1f5fa15bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:27.576529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59915bff-b05f-4cd9-94fd-08c1f5fa15bd service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:27.576803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59915bff-b05f-4cd9-94fd-08c1f5fa15bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:27.577066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59915bff-b05f-4cd9-94fd-08c1f5fa15bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:27.578726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-59915bff-b05f-4cd9-94fd-08c1f5fa15bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:27.685668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59915bff-b05f-4cd9-94fd-08c1f5fa15bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:27.687007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59915bff-b05f-4cd9-94fd-08c1f5fa15bd service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:27.708041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 738/1470] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:37.110257 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=22 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:37.705786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=66 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:47.892048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dafd632-89f3-4508-a568-ec7daecb1e09 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:47.892584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dafd632-89f3-4508-a568-ec7daecb1e09 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:47.892584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dafd632-89f3-4508-a568-ec7daecb1e09 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:47.893928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 733/1471] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:48:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:47.899214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a30704-9b04-4b83-8ce3-4fc35459222f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:47.900253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a30704-9b04-4b83-8ce3-4fc35459222f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:47.900625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a30704-9b04-4b83-8ce3-4fc35459222f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:47.965007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e5a30704-9b04-4b83-8ce3-4fc35459222f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:47.966189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e5a30704-9b04-4b83-8ce3-4fc35459222f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:48.062538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 739/1472] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:48:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:48.369723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e9ec7e-8c71-4f3e-9806-d043f2a09f89 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:48.370003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e9ec7e-8c71-4f3e-9806-d043f2a09f89 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:48.370274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e9ec7e-8c71-4f3e-9806-d043f2a09f89 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:48.371576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 734/1473] 10.222.0.42 () {54 vars in 962 bytes} [Fri Sep 16 01:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:48.376521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17205d5-66ac-4e9e-9fe0-06d4ad594f3a None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:48.376803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17205d5-66ac-4e9e-9fe0-06d4ad594f3a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:48.377071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17205d5-66ac-4e9e-9fe0-06d4ad594f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:48.455890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f17205d5-66ac-4e9e-9fe0-06d4ad594f3a None None] MFA Rules not processed for user `30b10ebdc87a4747bef7fc60b3cec9c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:48.457118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f17205d5-66ac-4e9e-9fe0-06d4ad594f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:48.477257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:48.532559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 740/1474] 10.222.0.42 () {58 vars in 1059 bytes} [Fri Sep 16 01:48:48 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:48:48.540592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e9ec7e-8c71-4f3e-9806-d043f2a09f89 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:48.542224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-80e9ec7e-8c71-4f3e-9806-d043f2a09f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:48.646411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e9ec7e-8c71-4f3e-9806-d043f2a09f89 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:48.721055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64deed5d-46e8-4e95-9dda-73470a18d32d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:48.721648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64deed5d-46e8-4e95-9dda-73470a18d32d service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:48.721925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64deed5d-46e8-4e95-9dda-73470a18d32d service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:48.722193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64deed5d-46e8-4e95-9dda-73470a18d32d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:48.723774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-64deed5d-46e8-4e95-9dda-73470a18d32d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:48.782099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:48.836580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64deed5d-46e8-4e95-9dda-73470a18d32d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:48.838671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64deed5d-46e8-4e95-9dda-73470a18d32d service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:48.856893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 735/1475] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:51.128660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8b335-76a1-48eb-ba00-126fdcf97c87 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:51.129250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8b335-76a1-48eb-ba00-126fdcf97c87 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:51.129250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b8b335-76a1-48eb-ba00-126fdcf97c87 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:51.130542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 741/1476] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:48:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:51.137193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e872a-ed01-4d0c-97e9-4e9be3ee7f50 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:51.137611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e872a-ed01-4d0c-97e9-4e9be3ee7f50 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:51.137740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e872a-ed01-4d0c-97e9-4e9be3ee7f50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:51.204180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-dd5e872a-ed01-4d0c-97e9-4e9be3ee7f50 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:51.205412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dd5e872a-ed01-4d0c-97e9-4e9be3ee7f50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:51.300736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 736/1477] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:48:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:51.576365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8b335-76a1-48eb-ba00-126fdcf97c87 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:51.577861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-72b8b335-76a1-48eb-ba00-126fdcf97c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:51.701644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b8b335-76a1-48eb-ba00-126fdcf97c87 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:51.751862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:51.795802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac7d84f-6a07-4b10-8e4a-321dc6992c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:51.796443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac7d84f-6a07-4b10-8e4a-321dc6992c61 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:51.796735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac7d84f-6a07-4b10-8e4a-321dc6992c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:51.796991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac7d84f-6a07-4b10-8e4a-321dc6992c61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:51.797622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac7d84f-6a07-4b10-8e4a-321dc6992c61 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:51.799356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-eac7d84f-6a07-4b10-8e4a-321dc6992c61 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:48:51.799356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:48:51.799356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:48:51.800516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac7d84f-6a07-4b10-8e4a-321dc6992c61 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:51.820659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 742/1478] 10.222.0.42 () {56 vars in 1108 bytes} [Fri Sep 16 01:48:51 2022] GET /identity/v3/projects => generated 2553 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:48:53.982015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c50b3-a32e-42cf-8d09-3c10e51179d3 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:53.982586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c50b3-a32e-42cf-8d09-3c10e51179d3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:53.982586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438c50b3-a32e-42cf-8d09-3c10e51179d3 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:53.983959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 737/1479] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:48:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:53.989336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630a6bb-fd8f-43c7-9c4e-0931ef9ce4a6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:53.989783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630a6bb-fd8f-43c7-9c4e-0931ef9ce4a6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:53.989886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630a6bb-fd8f-43c7-9c4e-0931ef9ce4a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:54.055324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7630a6bb-fd8f-43c7-9c4e-0931ef9ce4a6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:54.056602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7630a6bb-fd8f-43c7-9c4e-0931ef9ce4a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:54.151204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 743/1480] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:48:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:54.450035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c50b3-a32e-42cf-8d09-3c10e51179d3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:54.451807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-438c50b3-a32e-42cf-8d09-3c10e51179d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:54.461506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:54.595485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438c50b3-a32e-42cf-8d09-3c10e51179d3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:54.674864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:54.708847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4013801-ea43-4b51-8124-92a645569f63 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:54.709380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4013801-ea43-4b51-8124-92a645569f63 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:54.709651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4013801-ea43-4b51-8124-92a645569f63 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:54.709911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4013801-ea43-4b51-8124-92a645569f63 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:54.711522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e4013801-ea43-4b51-8124-92a645569f63 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:54.802354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4013801-ea43-4b51-8124-92a645569f63 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:54.803959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4013801-ea43-4b51-8124-92a645569f63 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:54.823179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 738/1481] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:48:57.153383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd967d7a-30c7-475a-b0c2-263087060ab9 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:57.153962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd967d7a-30c7-475a-b0c2-263087060ab9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:57.153962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd967d7a-30c7-475a-b0c2-263087060ab9 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:57.155258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 744/1482] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:48:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:48:57.160610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed5b23-45b6-43d0-a1f9-e497e5144665 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:57.161023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed5b23-45b6-43d0-a1f9-e497e5144665 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:57.161161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed5b23-45b6-43d0-a1f9-e497e5144665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:57.228726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-79ed5b23-45b6-43d0-a1f9-e497e5144665 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:48:57.229943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-79ed5b23-45b6-43d0-a1f9-e497e5144665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:57.277464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:57.327962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 739/1483] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:48:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:48:57.622951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd967d7a-30c7-475a-b0c2-263087060ab9 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:48:57.624531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fd967d7a-30c7-475a-b0c2-263087060ab9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:57.649716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:57.730088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd967d7a-30c7-475a-b0c2-263087060ab9 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:48:57.807420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4591bf9d-c6c3-46fb-bed4-aad5f031332f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:48:57.808045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4591bf9d-c6c3-46fb-bed4-aad5f031332f service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:48:57.808318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4591bf9d-c6c3-46fb-bed4-aad5f031332f service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:48:57.808582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4591bf9d-c6c3-46fb-bed4-aad5f031332f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:48:57.810197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4591bf9d-c6c3-46fb-bed4-aad5f031332f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:48:57.886035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:48:57.920945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4591bf9d-c6c3-46fb-bed4-aad5f031332f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:48:57.922376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4591bf9d-c6c3-46fb-bed4-aad5f031332f service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:48:57.940090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 745/1484] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:48:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:49:00.150690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a2d01-3172-4cc0-9eef-bea8768eff20 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:00.151269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a2d01-3172-4cc0-9eef-bea8768eff20 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:00.151269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8a2d01-3172-4cc0-9eef-bea8768eff20 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:00.152600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 740/1485] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:49:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:49:00.157792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bf052c-da31-460e-9a90-74bcf4ec6bcd None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:00.158218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bf052c-da31-460e-9a90-74bcf4ec6bcd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:00.158344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bf052c-da31-460e-9a90-74bcf4ec6bcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:00.223617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d1bf052c-da31-460e-9a90-74bcf4ec6bcd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:49:00.224902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d1bf052c-da31-460e-9a90-74bcf4ec6bcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:00.320396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 746/1486] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:49:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:49:00.614474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8a2d01-3172-4cc0-9eef-bea8768eff20 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:49:00.616040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cd8a2d01-3172-4cc0-9eef-bea8768eff20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:00.723838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8a2d01-3172-4cc0-9eef-bea8768eff20 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:49:00.799403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:49:00.799403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d94ac9-4208-4cc1-b2c0-f55aa4d0d2db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:49:00.799403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d94ac9-4208-4cc1-b2c0-f55aa4d0d2db service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:00.799403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d94ac9-4208-4cc1-b2c0-f55aa4d0d2db service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:00.799889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d94ac9-4208-4cc1-b2c0-f55aa4d0d2db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:00.800548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-48d94ac9-4208-4cc1-b2c0-f55aa4d0d2db service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:00.911405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d94ac9-4208-4cc1-b2c0-f55aa4d0d2db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:49:00.912799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d94ac9-4208-4cc1-b2c0-f55aa4d0d2db service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:49:00.930644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 741/1487] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:49:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:49:03.233972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029202b7-44c4-4dd6-92bc-7562b2399810 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:03.234499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029202b7-44c4-4dd6-92bc-7562b2399810 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:03.234499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029202b7-44c4-4dd6-92bc-7562b2399810 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:03.235835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 747/1488] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:49:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:49:03.241184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e3222f-e605-469b-9620-61a6193c6546 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:03.241601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e3222f-e605-469b-9620-61a6193c6546 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:03.241845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e3222f-e605-469b-9620-61a6193c6546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:03.310249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-80e3222f-e605-469b-9620-61a6193c6546 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:49:03.311447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-80e3222f-e605-469b-9620-61a6193c6546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:03.362430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:49:03.408504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 742/1489] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:49:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:49:03.705210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029202b7-44c4-4dd6-92bc-7562b2399810 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:49:03.706730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-029202b7-44c4-4dd6-92bc-7562b2399810 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:03.742637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:49:03.810912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029202b7-44c4-4dd6-92bc-7562b2399810 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:49:03.885844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63320223-77f6-460f-9569-03b4d7b76c40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:49:03.886444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63320223-77f6-460f-9569-03b4d7b76c40 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:03.886720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63320223-77f6-460f-9569-03b4d7b76c40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:03.886985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63320223-77f6-460f-9569-03b4d7b76c40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:03.888580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-63320223-77f6-460f-9569-03b4d7b76c40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:03.966885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:49:03.996681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63320223-77f6-460f-9569-03b4d7b76c40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:49:03.998111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63320223-77f6-460f-9569-03b4d7b76c40 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:49:04.016927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 748/1490] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:49:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:49:06.492773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1980c8-922d-4186-b31d-569def39ba31 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:06.492773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1980c8-922d-4186-b31d-569def39ba31 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:06.493940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1980c8-922d-4186-b31d-569def39ba31 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:06.494671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 743/1491] 10.222.0.42 () {54 vars in 927 bytes} [Fri Sep 16 01:49:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:49:06.499923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b570c-9c84-44c5-93a2-7130f3eb56cc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:06.500391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b570c-9c84-44c5-93a2-7130f3eb56cc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:06.500500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b570c-9c84-44c5-93a2-7130f3eb56cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:06.568448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-801b570c-9c84-44c5-93a2-7130f3eb56cc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:49:06.569642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-801b570c-9c84-44c5-93a2-7130f3eb56cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:06.666243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 749/1492] 10.222.0.42 () {58 vars in 1024 bytes} [Fri Sep 16 01:49:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:49:06.958000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1980c8-922d-4186-b31d-569def39ba31 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:49:06.959533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1d1980c8-922d-4186-b31d-569def39ba31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:07.063716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1980c8-922d-4186-b31d-569def39ba31 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:49:07.136785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f2710b-848f-4016-ade9-9a55eecb4e2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:49:07.137385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f2710b-848f-4016-ade9-9a55eecb4e2f service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:49:07.137660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f2710b-848f-4016-ade9-9a55eecb4e2f service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:49:07.137926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f2710b-848f-4016-ade9-9a55eecb4e2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:49:07.139507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-99f2710b-848f-4016-ade9-9a55eecb4e2f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:49:07.147859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:49:07.364076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f2710b-848f-4016-ade9-9a55eecb4e2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:49:07.365509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f2710b-848f-4016-ade9-9a55eecb4e2f service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:49:07.385044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 744/1493] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:49:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:49:16.664201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=82 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:49:17.381804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=50 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:19.864324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:19.864921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:19.864921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:19.940226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:50:19.941423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:20.038196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 750/1494] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:50:20.046141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6b86e-b8bf-4ab8-b3fc-ac55668dbf87 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:20.046622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6b86e-b8bf-4ab8-b3fc-ac55668dbf87 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:20.046730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6b86e-b8bf-4ab8-b3fc-ac55668dbf87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:20.123446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-20e6b86e-b8bf-4ab8-b3fc-ac55668dbf87 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:50:20.124665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-20e6b86e-b8bf-4ab8-b3fc-ac55668dbf87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:20.226448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 745/1495] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:50:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:50:20.231033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:20.232243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:20.244325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974f1c76-2c66-430f-aed5-005fc7689b8d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:20.304982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:20.338066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:20.338661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:20.338938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:20.339204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:20.339832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:50:20.340166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:20.341159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:20.341159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:50:20.341159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:50:20.342035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:20.353654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 751/1496] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:50:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:50:20.361941 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e6b86e-b8bf-4ab8-b3fc-ac55668dbf87 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:20.371351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e6b86e-b8bf-4ab8-b3fc-ac55668dbf87 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:20.432140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:20.465894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:20.466495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:20.466773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:20.467042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:20.467675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:20.469450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:20.469450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 16 01:50:20.469450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: Sep 16 01:50:20.470631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:20.526858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 746/1497] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:50:20 2022] POST /identity/v3/projects => generated 377 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:50:20.531067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:20.532588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:20.566586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f46af1b-4e25-4db7-b5db-e9ed36204f96 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:20.666541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:20.667145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:20.667425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:20.667696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:20.668358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:20.669363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:20.669363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:50:20.669363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:50:20.670577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:20.712897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 752/1498] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:50:20 2022] POST /identity/v3/users => generated 289 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:50:20.717028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:20.718615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:20.779136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a36f33-e835-4830-be71-54685dd491ee admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:20.865763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:20.880059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:20.880863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:20.881267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:20.881659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:20.882762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:20.883787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:20.883787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 16 01:50:20.883787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: Sep 16 01:50:20.884694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:20.898449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 747/1499] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:50:20.902687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:20.904218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:20.916031 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4689bd-e38b-4953-9e1e-e26f98e939b1 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:20.940951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:21.012389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ec1b07-33af-4948-b58b-c9f0f90e2ead admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:21.012912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec1b07-33af-4948-b58b-c9f0f90e2ead admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:21.013185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec1b07-33af-4948-b58b-c9f0f90e2ead admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:21.013450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec1b07-33af-4948-b58b-c9f0f90e2ead admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:21.014078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ec1b07-33af-4948-b58b-c9f0f90e2ead admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:21.015075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-06ec1b07-33af-4948-b58b-c9f0f90e2ead admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:21.015075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:50:21.015075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:50:21.015963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ec1b07-33af-4948-b58b-c9f0f90e2ead admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:21.030471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 753/1500] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:50:21.034527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:21.036647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:21.069778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462e2504-3fc2-46fb-a176-141ab0902c9c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:21.163911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a5b7281-a218-4260-a3e8-152a2edf96ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:21.164582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b7281-a218-4260-a3e8-152a2edf96ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:21.164857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b7281-a218-4260-a3e8-152a2edf96ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:21.165121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a5b7281-a218-4260-a3e8-152a2edf96ed admin admin] PATH_INFO: `/v3/projects/4b85c29ea4bd44a590642f8286f01741/users/12599b9b44d84c7f8f9db6027b955ee6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:21.187549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5b7281-a218-4260-a3e8-152a2edf96ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b85c29ea4bd44a590642f8286f01741, user_id=12599b9b44d84c7f8f9db6027b955ee6, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:21.188587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-3a5b7281-a218-4260-a3e8-152a2edf96ed admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:21.188587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 16 01:50:21.188587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: Sep 16 01:50:21.190503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a5b7281-a218-4260-a3e8-152a2edf96ed admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:21.215171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 748/1501] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:50:21 2022] PUT /identity/v3/projects/4b85c29ea4bd44a590642f8286f01741/users/12599b9b44d84c7f8f9db6027b955ee6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:50:21.227349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb79b656-42e8-4551-86e0-f6d913ac2df4 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:21.227635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb79b656-42e8-4551-86e0-f6d913ac2df4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:21.227907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb79b656-42e8-4551-86e0-f6d913ac2df4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:21.284807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-fb79b656-42e8-4551-86e0-f6d913ac2df4 None None] MFA Rules not processed for user `12599b9b44d84c7f8f9db6027b955ee6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:50:21.286120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fb79b656-42e8-4551-86e0-f6d913ac2df4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:21.362724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 754/1502] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:50:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:50:21.376381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97970173-88bc-439e-8fb5-c7e36eeb453e None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:21.376663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97970173-88bc-439e-8fb5-c7e36eeb453e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:21.376932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97970173-88bc-439e-8fb5-c7e36eeb453e None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:21.378241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 749/1503] 10.222.0.42 () {54 vars in 964 bytes} [Fri Sep 16 01:50:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:50:21.384302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0751e274-71b1-4e1e-9b3d-89dba4f44173 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:21.384585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0751e274-71b1-4e1e-9b3d-89dba4f44173 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:21.384852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0751e274-71b1-4e1e-9b3d-89dba4f44173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:21.415391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:21.464361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0751e274-71b1-4e1e-9b3d-89dba4f44173 None None] MFA Rules not processed for user `d7b63222dc2846f5a42208877b0de123`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:50:21.465565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0751e274-71b1-4e1e-9b3d-89dba4f44173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:21.541237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 755/1504] 10.222.0.42 () {58 vars in 1061 bytes} [Fri Sep 16 01:50:21 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:50:21.548967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97970173-88bc-439e-8fb5-c7e36eeb453e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:21.550520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-97970173-88bc-439e-8fb5-c7e36eeb453e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:21.601458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:21.655666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97970173-88bc-439e-8fb5-c7e36eeb453e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:21.726953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:21.727546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:21.727824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:21.728123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:21.729713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:21.828501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:21.829927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:21.833914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:21.848714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 750/1505] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:50:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:50:21.894273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670d16ac-6967-4f86-951e-b6578998cca6 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:21.894561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670d16ac-6967-4f86-951e-b6578998cca6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:21.894839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670d16ac-6967-4f86-951e-b6578998cca6 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:21.896736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 756/1506] 10.222.0.42 () {52 vars in 830 bytes} [Fri Sep 16 01:50:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:50:21.900218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:21.901408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:21.998273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92a71eb-a40e-4f01-9c5c-f7cd705516e4 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:22.089942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:22.090538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:22.090746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:22.091009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] PATH_INFO: `/v3/users/12599b9b44d84c7f8f9db6027b955ee6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:22.103433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] RBAC: Authorizing `identity:delete_user(user_id=12599b9b44d84c7f8f9db6027b955ee6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:22.104480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:22.104480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 16 01:50:22.104480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: Sep 16 01:50:22.105684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:22.116186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:22.172401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12599b9b44d84c7f8f9db6027b955ee6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:22.172832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12599b9b44d84c7f8f9db6027b955ee6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:22.186814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12599b9b44d84c7f8f9db6027b955ee6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:22.187364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12599b9b44d84c7f8f9db6027b955ee6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:22.210985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12599b9b44d84c7f8f9db6027b955ee6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:22.226121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 751/1507] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:50:21 2022] DELETE /identity/v3/users/12599b9b44d84c7f8f9db6027b955ee6 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:50:22.230533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:22.231012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:22.231123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:22.289406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:50:22.290622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:22.367084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:22.380699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 757/1508] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:50:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:50:22.390547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:22.391748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:22.493089 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c041ec0d-f0b2-4a19-b24c-db99880d4adc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:22.546547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:22.567997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a3b8d-6683-4304-9cf5-ca253fde90d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:22.568604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a3b8d-6683-4304-9cf5-ca253fde90d6 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:22.568884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a3b8d-6683-4304-9cf5-ca253fde90d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:22.569151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a3b8d-6683-4304-9cf5-ca253fde90d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:22.570744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-745a3b8d-6683-4304-9cf5-ca253fde90d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:22.678188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a3b8d-6683-4304-9cf5-ca253fde90d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:22.679632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a3b8d-6683-4304-9cf5-ca253fde90d6 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:22.697763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 752/1509] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:50:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:50:23.056549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:50:23.057759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:50:23.069622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531f32ee-5063-4671-bd94-92b1f71d934f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:50:23.166332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:50:23.166956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:50:23.167234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:50:23.167503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] PATH_INFO: `/v3/projects/4b85c29ea4bd44a590642f8286f01741` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:50:23.178325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b85c29ea4bd44a590642f8286f01741)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:50:23.179347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:50:23.179347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:50:23.179347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:50:23.180596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:50:23.239481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b85c29ea4bd44a590642f8286f01741', 'request_id': 'req-66bcde68-35d6-4f85-88e7-45f7d4b48a34'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:23.254978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b85c29ea4bd44a590642f8286f01741', 'request_id': 'req-66bcde68-35d6-4f85-88e7-45f7d4b48a34'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:23.261402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b85c29ea4bd44a590642f8286f01741', 'request_id': 'req-66bcde68-35d6-4f85-88e7-45f7d4b48a34'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:23.287481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] The token cache is being invalidate because project 4b85c29ea4bd44a590642f8286f01741 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:50:23.287766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:23.288306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66bcde68-35d6-4f85-88e7-45f7d4b48a34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:50:23.289594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 758/1510] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:50:23 2022] DELETE /identity/v3/projects/4b85c29ea4bd44a590642f8286f01741 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:50:32.695539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=62 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:50:33.284454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:51:50.593834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee966547-9085-49af-ae0b-8f81563c381d None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:50.594451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee966547-9085-49af-ae0b-8f81563c381d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:50.594451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee966547-9085-49af-ae0b-8f81563c381d None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:50.595731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 753/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:51:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:51:50.600736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac83583-b2fa-42fa-96a7-7e91d1162835 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:50.601213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac83583-b2fa-42fa-96a7-7e91d1162835 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:50.601310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac83583-b2fa-42fa-96a7-7e91d1162835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:50.688895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-2ac83583-b2fa-42fa-96a7-7e91d1162835 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:51:50.690108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2ac83583-b2fa-42fa-96a7-7e91d1162835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:51:50.794468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 759/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:51:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:51:50.947724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee966547-9085-49af-ae0b-8f81563c381d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:51:50.950053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ee966547-9085-49af-ae0b-8f81563c381d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:51:51.070077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee966547-9085-49af-ae0b-8f81563c381d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:51:51.075708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d81b8e1-32a8-41ec-bc7a-bc96b1a443fe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:51:51.076341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d81b8e1-32a8-41ec-bc7a-bc96b1a443fe service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:51.076620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d81b8e1-32a8-41ec-bc7a-bc96b1a443fe service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:51.076891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d81b8e1-32a8-41ec-bc7a-bc96b1a443fe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:51.084969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d81b8e1-32a8-41ec-bc7a-bc96b1a443fe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:51:51.086555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d81b8e1-32a8-41ec-bc7a-bc96b1a443fe service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:51:51.096016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 754/1513] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:51:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:51:52.187852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8963e9-999f-4460-88b2-52a801f4307c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:52.188445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8963e9-999f-4460-88b2-52a801f4307c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:52.188445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8963e9-999f-4460-88b2-52a801f4307c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:52.189827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 760/1514] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:51:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:51:52.197223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f4509-2bf0-43c5-940f-bb03007d6850 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:52.197737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f4509-2bf0-43c5-940f-bb03007d6850 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:52.197852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f4509-2bf0-43c5-940f-bb03007d6850 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:52.216800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-396f4509-2bf0-43c5-940f-bb03007d6850 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:51:52.217988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-396f4509-2bf0-43c5-940f-bb03007d6850 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:51:52.225642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 755/1515] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:51:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:51:52.386646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8963e9-999f-4460-88b2-52a801f4307c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:51:52.401073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8963e9-999f-4460-88b2-52a801f4307c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:51:52.406643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a64a7a-ede5-43e2-9b21-d35968e4d7eb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:51:52.407231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a64a7a-ede5-43e2-9b21-d35968e4d7eb service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:52.407511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a64a7a-ede5-43e2-9b21-d35968e4d7eb service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:52.407775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a64a7a-ede5-43e2-9b21-d35968e4d7eb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:52.416434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a64a7a-ede5-43e2-9b21-d35968e4d7eb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:51:52.417993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a64a7a-ede5-43e2-9b21-d35968e4d7eb service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:51:52.428197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 761/1516] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:51:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:51:52.517562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b7fc02-2fa0-4c54-bf93-ea84656875b1 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:52.517882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b7fc02-2fa0-4c54-bf93-ea84656875b1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:52.518153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b7fc02-2fa0-4c54-bf93-ea84656875b1 None None] PATH_INFO: `/v3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:52.519626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 756/1517] 10.222.0.50 () {54 vars in 925 bytes} [Fri Sep 16 01:51:52 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:51:52.527215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c97331-514b-40ca-b52a-5f582d03a7bf None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:51:52.527515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c97331-514b-40ca-b52a-5f582d03a7bf None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:51:52.527752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c97331-514b-40ca-b52a-5f582d03a7bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:51:52.596878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-70c97331-514b-40ca-b52a-5f582d03a7bf None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:51:52.598976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-70c97331-514b-40ca-b52a-5f582d03a7bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:51:52.616859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:51:52.674078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 762/1518] 10.222.0.50 () {58 vars in 1015 bytes} [Fri Sep 16 01:51:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:01.571995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14dd2b-c49a-45d4-b2b6-6d0e1beada18 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:01.572471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14dd2b-c49a-45d4-b2b6-6d0e1beada18 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:01.572567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a14dd2b-c49a-45d4-b2b6-6d0e1beada18 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:01.574275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 757/1519] 10.222.0.42 () {54 vars in 961 bytes} [Fri Sep 16 01:52:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:52:01.579373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3dac83-7293-4bd7-8d12-4779a6a235f2 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:01.579787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3dac83-7293-4bd7-8d12-4779a6a235f2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:01.579911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3dac83-7293-4bd7-8d12-4779a6a235f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:01.599630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-9a3dac83-7293-4bd7-8d12-4779a6a235f2 None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:52:01.600927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9a3dac83-7293-4bd7-8d12-4779a6a235f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:01.609780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 763/1520] 10.222.0.42 () {58 vars in 1058 bytes} [Fri Sep 16 01:52:01 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:01.617219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a14dd2b-c49a-45d4-b2b6-6d0e1beada18 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:52:01.630331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a14dd2b-c49a-45d4-b2b6-6d0e1beada18 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:52:01.635716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4569c9b-953c-4cb2-8c21-e705b82be9b4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:52:01.636324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4569c9b-953c-4cb2-8c21-e705b82be9b4 service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:01.636600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4569c9b-953c-4cb2-8c21-e705b82be9b4 service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:01.636865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4569c9b-953c-4cb2-8c21-e705b82be9b4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:01.644908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4569c9b-953c-4cb2-8c21-e705b82be9b4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:52:01.646464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4569c9b-953c-4cb2-8c21-e705b82be9b4 service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:52:01.656880 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 758/1521] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:52:01 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:52:03.908864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34059363-eed0-4b60-a642-7635c2d66878 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:03.909428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34059363-eed0-4b60-a642-7635c2d66878 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:03.909428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34059363-eed0-4b60-a642-7635c2d66878 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:03.910737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 764/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:52:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:52:03.917627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e80f85-46b6-4a43-bce7-1d3ab1260323 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:03.918054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e80f85-46b6-4a43-bce7-1d3ab1260323 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:03.918180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e80f85-46b6-4a43-bce7-1d3ab1260323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:03.937039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-98e80f85-46b6-4a43-bce7-1d3ab1260323 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:52:03.938220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-98e80f85-46b6-4a43-bce7-1d3ab1260323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:03.946140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 759/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:52:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:04.099457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34059363-eed0-4b60-a642-7635c2d66878 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:52:04.114302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34059363-eed0-4b60-a642-7635c2d66878 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:52:04.120002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88b8f80-19fc-44cf-b070-c9db825be187 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:52:04.120584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b8f80-19fc-44cf-b070-c9db825be187 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:04.120857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b8f80-19fc-44cf-b070-c9db825be187 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:04.121123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88b8f80-19fc-44cf-b070-c9db825be187 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:04.128968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88b8f80-19fc-44cf-b070-c9db825be187 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:52:04.130516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88b8f80-19fc-44cf-b070-c9db825be187 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:52:04.140406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 765/1524] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:52:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:52:05.214267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11dde41-517a-404a-a249-0f87f6a2399f None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:05.214837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11dde41-517a-404a-a249-0f87f6a2399f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:05.214837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11dde41-517a-404a-a249-0f87f6a2399f None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:05.216185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 760/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 16 01:52:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:52:05.222400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a443dcc-ae68-4410-aa73-b49cd99f4a6a None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:05.222811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a443dcc-ae68-4410-aa73-b49cd99f4a6a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:05.222951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a443dcc-ae68-4410-aa73-b49cd99f4a6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:05.242121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7a443dcc-ae68-4410-aa73-b49cd99f4a6a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:52:05.243324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7a443dcc-ae68-4410-aa73-b49cd99f4a6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:05.251099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 766/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 16 01:52:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:05.403301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11dde41-517a-404a-a249-0f87f6a2399f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:52:05.418025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11dde41-517a-404a-a249-0f87f6a2399f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:52:05.423336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24ab88b-d1e2-40fb-abae-c9313272a72e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:52:05.423947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24ab88b-d1e2-40fb-abae-c9313272a72e service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:05.424216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24ab88b-d1e2-40fb-abae-c9313272a72e service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:05.424479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24ab88b-d1e2-40fb-abae-c9313272a72e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:05.433152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24ab88b-d1e2-40fb-abae-c9313272a72e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:52:05.434704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24ab88b-d1e2-40fb-abae-c9313272a72e service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:52:05.443776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 761/1527] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:52:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:52:09.433883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a73f4-e771-408e-9110-5851e58ebb2d None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:09.434455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a73f4-e771-408e-9110-5851e58ebb2d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:09.434455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650a73f4-e771-408e-9110-5851e58ebb2d None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:09.435818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 767/1528] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:52:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:52:09.440900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510bd81-cdca-4291-9c76-88b26ceb9b2c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:09.441339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510bd81-cdca-4291-9c76-88b26ceb9b2c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:09.441453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510bd81-cdca-4291-9c76-88b26ceb9b2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:09.463480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7510bd81-cdca-4291-9c76-88b26ceb9b2c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:52:09.464803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7510bd81-cdca-4291-9c76-88b26ceb9b2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:09.474299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 762/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:52:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:09.768779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650a73f4-e771-408e-9110-5851e58ebb2d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:52:09.770433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-650a73f4-e771-408e-9110-5851e58ebb2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:09.850333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650a73f4-e771-408e-9110-5851e58ebb2d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:52:09.854514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e831492f-3b26-4213-9861-fe3ee110dc01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:52:09.855107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e831492f-3b26-4213-9861-fe3ee110dc01 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:09.855384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e831492f-3b26-4213-9861-fe3ee110dc01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:09.855652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e831492f-3b26-4213-9861-fe3ee110dc01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:09.864431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e831492f-3b26-4213-9861-fe3ee110dc01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:52:09.865841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e831492f-3b26-4213-9861-fe3ee110dc01 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:52:09.875370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 768/1530] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:52:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:52:12.020390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724cfa0-651c-4292-9af5-6ad38fa0e9a3 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:12.020922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724cfa0-651c-4292-9af5-6ad38fa0e9a3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:12.020922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6724cfa0-651c-4292-9af5-6ad38fa0e9a3 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:12.023114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 763/1531] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:52:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:52:12.028391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f316c-36b8-4f1a-9f8c-39e909b72003 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:12.028826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f316c-36b8-4f1a-9f8c-39e909b72003 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:12.028973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f316c-36b8-4f1a-9f8c-39e909b72003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:12.049222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b19f316c-36b8-4f1a-9f8c-39e909b72003 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:52:12.050414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b19f316c-36b8-4f1a-9f8c-39e909b72003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:12.058431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 769/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:52:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:12.358334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724cfa0-651c-4292-9af5-6ad38fa0e9a3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:52:12.369522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6724cfa0-651c-4292-9af5-6ad38fa0e9a3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:52:12.373041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f67c43-0e83-4847-98e7-e40a23e41954 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:52:12.373625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f67c43-0e83-4847-98e7-e40a23e41954 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:12.373902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f67c43-0e83-4847-98e7-e40a23e41954 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:12.374171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f67c43-0e83-4847-98e7-e40a23e41954 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:12.383758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f67c43-0e83-4847-98e7-e40a23e41954 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:52:12.385277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f67c43-0e83-4847-98e7-e40a23e41954 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:52:12.394591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 764/1533] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:52:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:52:15.969704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b93f0-b9e8-402a-9216-608729fff7c3 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:15.970224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b93f0-b9e8-402a-9216-608729fff7c3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:15.970352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657b93f0-b9e8-402a-9216-608729fff7c3 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:15.971591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 770/1534] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:52:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:52:15.977623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74de13-32b0-4e6b-b17e-ff5eb453431e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:15.978077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74de13-32b0-4e6b-b17e-ff5eb453431e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:15.978190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac74de13-32b0-4e6b-b17e-ff5eb453431e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:15.985784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=81 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:52:15.998708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ac74de13-32b0-4e6b-b17e-ff5eb453431e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:52:15.999905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ac74de13-32b0-4e6b-b17e-ff5eb453431e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:16.008809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 765/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:52:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:16.313970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b93f0-b9e8-402a-9216-608729fff7c3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:52:16.315640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-657b93f0-b9e8-402a-9216-608729fff7c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:16.380260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:52:16.394894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657b93f0-b9e8-402a-9216-608729fff7c3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:52:16.398515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcd713c-0101-4c88-ae6b-792e49f243f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:52:16.399113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd713c-0101-4c88-ae6b-792e49f243f0 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:16.399391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd713c-0101-4c88-ae6b-792e49f243f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:16.399660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcd713c-0101-4c88-ae6b-792e49f243f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:16.409465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcd713c-0101-4c88-ae6b-792e49f243f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:52:16.410880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcd713c-0101-4c88-ae6b-792e49f243f0 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:52:16.420526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 771/1536] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:52:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:52:18.602384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b672b5-a16f-4d47-a74a-585706abea1e None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:18.602883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b672b5-a16f-4d47-a74a-585706abea1e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:18.602994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b672b5-a16f-4d47-a74a-585706abea1e None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:18.604408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 766/1537] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 16 01:52:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:52:18.610128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65883edd-1c45-4ca3-87eb-4428764c143e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:18.610555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65883edd-1c45-4ca3-87eb-4428764c143e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:18.610665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65883edd-1c45-4ca3-87eb-4428764c143e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:18.630051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-65883edd-1c45-4ca3-87eb-4428764c143e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:52:18.632029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-65883edd-1c45-4ca3-87eb-4428764c143e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:18.640056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 772/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 16 01:52:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:52:18.950564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b672b5-a16f-4d47-a74a-585706abea1e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:52:18.952266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f5b672b5-a16f-4d47-a74a-585706abea1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:52:18.965961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b672b5-a16f-4d47-a74a-585706abea1e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:52:18.969463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e9d1dc-93b3-4230-8472-9540ca8f284b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:52:18.970044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9d1dc-93b3-4230-8472-9540ca8f284b service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:52:18.970324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9d1dc-93b3-4230-8472-9540ca8f284b service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:52:18.970590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e9d1dc-93b3-4230-8472-9540ca8f284b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:52:18.979326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e9d1dc-93b3-4230-8472-9540ca8f284b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:52:18.980774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e9d1dc-93b3-4230-8472-9540ca8f284b service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:52:18.989956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 767/1539] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:52:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:52:28.625279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=11 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:52:28.985901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=8 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:43.545850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5d60cb-5fb5-49ec-b300-744b48d4428b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.548849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5d60cb-5fb5-49ec-b300-744b48d4428b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.548849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5d60cb-5fb5-49ec-b300-744b48d4428b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.585555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5f5d60cb-5fb5-49ec-b300-744b48d4428b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.587814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5f5d60cb-5fb5-49ec-b300-744b48d4428b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.600309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 773/1540] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:53:43.601632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dee370-674b-4bba-a462-846c9c65bd3b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.602531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dee370-674b-4bba-a462-846c9c65bd3b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.603043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dee370-674b-4bba-a462-846c9c65bd3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.610761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22b224-2455-45d5-9734-c363bee834cc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.610761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22b224-2455-45d5-9734-c363bee834cc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.610761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a22b224-2455-45d5-9734-c363bee834cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.634725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-57dee370-674b-4bba-a462-846c9c65bd3b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.636346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-57dee370-674b-4bba-a462-846c9c65bd3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.636783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6a22b224-2455-45d5-9734-c363bee834cc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.641216 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6a22b224-2455-45d5-9734-c363bee834cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.646854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 768/1541] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:53:43.648356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c8b0e8-126f-444d-8997-70fac7c457b1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.649712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c8b0e8-126f-444d-8997-70fac7c457b1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.650365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c8b0e8-126f-444d-8997-70fac7c457b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.651803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 774/1542] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.656169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26271f5f-ecaf-44cb-b73d-18eeed1067d9 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.656169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26271f5f-ecaf-44cb-b73d-18eeed1067d9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.656169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26271f5f-ecaf-44cb-b73d-18eeed1067d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.681224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-36c8b0e8-126f-444d-8997-70fac7c457b1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.682821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-36c8b0e8-126f-444d-8997-70fac7c457b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.688996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-26271f5f-ecaf-44cb-b73d-18eeed1067d9 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.692251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-26271f5f-ecaf-44cb-b73d-18eeed1067d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.695480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 769/1543] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.697887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c8b0e8-126f-444d-8997-70fac7c457b1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:43.714310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 775/1544] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.716043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be06a913-c268-47de-9c22-b84a2d9605ad None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.718484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be06a913-c268-47de-9c22-b84a2d9605ad None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.719010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be06a913-c268-47de-9c22-b84a2d9605ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.723062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c8b0e8-126f-444d-8997-70fac7c457b1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:43.731825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:43.734790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.735317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.735829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.738500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:43.739040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:43.740979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:53:43.740979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 16 01:53:43.740979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: Sep 16 01:53:43.741870 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1897099b-3c3a-4e80-a36e-b8303a1cd361 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:43.758992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-be06a913-c268-47de-9c22-b84a2d9605ad None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.760634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-be06a913-c268-47de-9c22-b84a2d9605ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.768253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 770/1545] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:43.769842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188de551-4802-4512-bfbc-a9434bd1bfb4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.770464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188de551-4802-4512-bfbc-a9434bd1bfb4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.770988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188de551-4802-4512-bfbc-a9434bd1bfb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.780131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 776/1546] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.782061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c71e2b-62f7-4cc5-a4f7-4f448ad1bdc6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.782061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c71e2b-62f7-4cc5-a4f7-4f448ad1bdc6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.782061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c71e2b-62f7-4cc5-a4f7-4f448ad1bdc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.794700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-188de551-4802-4512-bfbc-a9434bd1bfb4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.798332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-188de551-4802-4512-bfbc-a9434bd1bfb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.805560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 771/1547] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.807218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c31054-3010-484a-b499-5277decc043c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.807538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-86c71e2b-62f7-4cc5-a4f7-4f448ad1bdc6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.807844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c31054-3010-484a-b499-5277decc043c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.808307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c31054-3010-484a-b499-5277decc043c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.808784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-86c71e2b-62f7-4cc5-a4f7-4f448ad1bdc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.817647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 777/1548] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.819220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c71e2b-62f7-4cc5-a4f7-4f448ad1bdc6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:43.828622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c71e2b-62f7-4cc5-a4f7-4f448ad1bdc6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:43.831655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-60c31054-3010-484a-b499-5277decc043c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.832888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-60c31054-3010-484a-b499-5277decc043c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.833498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:43.834097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.834373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.834638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.835259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:43.836981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:53:43.836981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:53:43.836981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:53:43.838392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:43.842005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 772/1549] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.843382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b87bd21-bb51-4c6a-9d61-bdd62ada3faa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.843836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b87bd21-bb51-4c6a-9d61-bdd62ada3faa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.844146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b87bd21-bb51-4c6a-9d61-bdd62ada3faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.869852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7b87bd21-bb51-4c6a-9d61-bdd62ada3faa None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.871620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7b87bd21-bb51-4c6a-9d61-bdd62ada3faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.879091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 778/1550] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/projects => generated 385 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:43.879807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 773/1551] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.887456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb699e2-816d-4cd8-8b0c-462b8f4ea5c1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.887456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb699e2-816d-4cd8-8b0c-462b8f4ea5c1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.887456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb699e2-816d-4cd8-8b0c-462b8f4ea5c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.888204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:43.901883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-dfb699e2-816d-4cd8-8b0c-462b8f4ea5c1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:43.903096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dfb699e2-816d-4cd8-8b0c-462b8f4ea5c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:43.952527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d694f80c-ecfd-4b12-abfc-5671512c0393 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:43.957342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:43.957936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.958213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.958533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.959212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:43.959509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:43.961353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:43.963794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 774/1552] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:43.964505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb699e2-816d-4cd8-8b0c-462b8f4ea5c1 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:43.973814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb699e2-816d-4cd8-8b0c-462b8f4ea5c1 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:43.983017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 779/1553] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:43.983017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:43.986403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f37eb0-71cf-4ad6-8161-192735f119ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:43.986890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f37eb0-71cf-4ad6-8161-192735f119ba admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.987168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f37eb0-71cf-4ad6-8161-192735f119ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.987444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f37eb0-71cf-4ad6-8161-192735f119ba admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.988240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f37eb0-71cf-4ad6-8161-192735f119ba admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:43.988538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f37eb0-71cf-4ad6-8161-192735f119ba admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:43.988809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237bc668-abfd-4d0f-958f-c625faedbbaa admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:43.990171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f37eb0-71cf-4ad6-8161-192735f119ba admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:43.995309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74b4897-50cf-4933-805a-407bd3e89bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:43.995884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74b4897-50cf-4933-805a-407bd3e89bb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:43.996192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74b4897-50cf-4933-805a-407bd3e89bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:43.996464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74b4897-50cf-4933-805a-407bd3e89bb5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:43.997092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74b4897-50cf-4933-805a-407bd3e89bb5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:43.997388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74b4897-50cf-4933-805a-407bd3e89bb5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:43.999539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74b4897-50cf-4933-805a-407bd3e89bb5 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.002970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 775/1554] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:44.004212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ba26de-8feb-4fcd-8b76-12f8f6e79ecd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.004497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ba26de-8feb-4fcd-8b76-12f8f6e79ecd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.004783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ba26de-8feb-4fcd-8b76-12f8f6e79ecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.020144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 780/1555] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:44.020144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f0c51-227d-4831-a147-1f44ae80c17f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.020144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f0c51-227d-4831-a147-1f44ae80c17f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.020144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f0c51-227d-4831-a147-1f44ae80c17f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.024989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-54ba26de-8feb-4fcd-8b76-12f8f6e79ecd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:44.026194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-54ba26de-8feb-4fcd-8b76-12f8f6e79ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:44.033901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-396f0c51-227d-4831-a147-1f44ae80c17f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:44.035121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-396f0c51-227d-4831-a147-1f44ae80c17f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:44.037719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 776/1556] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:44.038516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ba26de-8feb-4fcd-8b76-12f8f6e79ecd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.044163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 781/1557] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:44.051879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f0e84b-1ff6-417b-b26b-83eb644696f6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.051879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f0e84b-1ff6-417b-b26b-83eb644696f6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.051879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f0e84b-1ff6-417b-b26b-83eb644696f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.052264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ba26de-8feb-4fcd-8b76-12f8f6e79ecd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.056526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.057123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.057397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.057663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.058285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.059328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:53:44.059328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 16 01:53:44.059328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: Sep 16 01:53:44.060628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.068729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a0f0e84b-1ff6-417b-b26b-83eb644696f6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:44.069975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a0f0e84b-1ff6-417b-b26b-83eb644696f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:44.078793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 782/1558] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:44.080179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470e4d4b-c0e5-4190-bdb2-06c9d69bd6d3 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.080475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470e4d4b-c0e5-4190-bdb2-06c9d69bd6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.080737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470e4d4b-c0e5-4190-bdb2-06c9d69bd6d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.085892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 777/1559] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/users => generated 293 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.086718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.096227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15500a9c-2d45-467c-bb60-ee167bb65983 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.100947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.101081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-470e4d4b-c0e5-4190-bdb2-06c9d69bd6d3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:44.102131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.102425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.102693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.102901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-470e4d4b-c0e5-4190-bdb2-06c9d69bd6d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:44.103463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.105408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.110661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 783/1560] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:44.111495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470e4d4b-c0e5-4190-bdb2-06c9d69bd6d3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.121803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470e4d4b-c0e5-4190-bdb2-06c9d69bd6d3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.126396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.126965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.127340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.127697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.128483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.130796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.138407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 778/1561] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/projects => generated 379 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.139204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.158200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 784/1562] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/projects => generated 389 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.158200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.197927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:44.200478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:44.203378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e71d0f1-a2a7-4e9c-89a2-84da6b00af56 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.225764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6260d6e-1079-4c64-a984-37bacee97c85 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.230691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70021dcc-82ad-4017-a4fa-aa20e71b5ff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.231298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70021dcc-82ad-4017-a4fa-aa20e71b5ff6 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.231579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70021dcc-82ad-4017-a4fa-aa20e71b5ff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.231853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70021dcc-82ad-4017-a4fa-aa20e71b5ff6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.232527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70021dcc-82ad-4017-a4fa-aa20e71b5ff6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:44.232826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70021dcc-82ad-4017-a4fa-aa20e71b5ff6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.234492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70021dcc-82ad-4017-a4fa-aa20e71b5ff6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.246535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 785/1563] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:44.247873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4564e18-5122-4c25-9469-c4885377d7b6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.248304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4564e18-5122-4c25-9469-c4885377d7b6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.248778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4564e18-5122-4c25-9469-c4885377d7b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.262135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.262848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.263133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.263407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.264094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.265998 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.269603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c4564e18-5122-4c25-9469-c4885377d7b6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:44.270800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c4564e18-5122-4c25-9469-c4885377d7b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:44.279431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 786/1564] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:44.280358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4564e18-5122-4c25-9469-c4885377d7b6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.291740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4564e18-5122-4c25-9469-c4885377d7b6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.297376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 779/1565] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/projects => generated 365 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.297376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.297624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.297624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.297624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.297911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.298328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:44.298622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.300231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.313178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 787/1566] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:44.314129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.328938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f487edf0-754d-4e39-87f1-5b0be0d35c0f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.362717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eb6aa6-b9dd-4c7b-af89-f8f355e2d41e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.367648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.368306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.368586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.368856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.369499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.371105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.381149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 780/1567] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:44.381944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.392534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e87f88-30af-4e09-9f4c-23aa091d08e2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.393720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.394395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.394759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.395075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.395788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:44.396180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.397142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.397715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.397980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.398086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.398251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.399440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.401490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.411533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 788/1568] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:44.412589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.423536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746df17a-bbf4-4b22-b2f3-0c559efb63cf admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.428176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 781/1569] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/users => generated 290 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.428176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.431833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.432452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.432729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.432996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.433644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.435532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.436312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75aa8c0-f6e1-4b35-88f6-188840b3db12 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.440858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.441433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.441710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.441975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.442585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.445121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.462229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 789/1570] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/users => generated 295 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.462229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.474092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1005ace7-36b1-4eec-9c25-d87d8915bb72 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.475478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 782/1571] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/projects => generated 377 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.476315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.521363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:44.534228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.534826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.535097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.535363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.536019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:53:44.536317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.537905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.539313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adc677b-c541-44d0-aebd-89404476c4e9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.544066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.544598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.544877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.545146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.545767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.547696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.551436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 790/1572] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:53:44.552269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.565622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63df826-3427-43a4-9fa3-26a028ea9819 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.570353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.571441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.571575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 783/1573] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/users => generated 283 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.571769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.572092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.572763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.574750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.575415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.585045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf12287-290d-4e96-8357-ed9eb24a6d3a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.590577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.591164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.591480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.591822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.592490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.594151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.594550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:44.603391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 791/1574] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/projects => generated 405 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.604516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.604703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 784/1575] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:44.605506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.615120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d483d8-ff3f-4eae-9350-4e4c014a9623 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.671851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9179dff6-4c17-4f01-997d-c5aaf96b59bd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.672942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.673613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.673889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.674157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.674794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.676425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.676659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.677259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.677545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.677813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.678474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.680385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.686874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 785/1576] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:44.687729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.698492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b57eef-e1ce-4070-a017-8ce956c8dba2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.703625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.704318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.704599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.704875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.705508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.707096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.709527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 792/1577] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/projects => generated 355 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.710337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.722190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 786/1578] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:44.722190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.729857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435838b7-1d08-47e4-8838-d7af73c33c17 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.780036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1978b61-9d9f-4ba2-b642-16116a41660a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.784832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.785605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.785888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.786156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.786770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.788717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.792657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.793259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.793538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.793804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.794429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.796359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.811411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 793/1579] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.812606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.823574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62400279-3a23-449e-8b32-1c0a55ae9504 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.830653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 787/1580] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/projects => generated 393 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.830653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.831585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.832209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.832486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.832752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.833412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.835039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.846718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 794/1581] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:44.850457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.860437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd14764e-70ef-4064-b842-4dcdb03521fe admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.864337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:44.893593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f56ebf-bd44-4457-95b9-a88fb22c16b9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.898537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.899130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.899415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.899683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.900329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.902255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.918797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.919480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.919756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.920059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] PATH_INFO: `/v3/projects/7b338918fdb447db92679939b9f534c4/users/2bd4786e1b444c318436ead072ca2541/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.923262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:44.926911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 788/1582] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/users => generated 303 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:44.928448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.936812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b338918fdb447db92679939b9f534c4, user_id=2bd4786e1b444c318436ead072ca2541, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.937838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:53:44.937838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:53:44.937838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:53:44.938279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5619a469-88cd-46c3-a0ff-52ea69306f75 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:44.939765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.943377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:44.943990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:44.944406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:44.944735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:44.946004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:44.948069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:44.953569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 795/1583] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:44 2022] PUT /identity/v3/projects/7b338918fdb447db92679939b9f534c4/users/2bd4786e1b444c318436ead072ca2541/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:44.960368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 789/1584] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:44.960498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.960584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:44.970530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc92430c-57b6-418a-968a-6ebcf8631c3f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.025318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3aa88-a430-4dc5-9886-73b8be086cbd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.028385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.028984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.029261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.029530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.030056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.030177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.030656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.030964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.031238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.031737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.031865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.033908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.041680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 790/1585] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.045079 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.055404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5446c9-e083-4af1-a8c0-dcd6878abb80 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.055893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 796/1586] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:44 2022] POST /identity/v3/users => generated 278 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:45.056646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.060458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.061080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.061550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.061823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.062459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.064092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.065979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41100473-5894-4e98-993e-d139cffa39ec admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.071266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.071854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.072164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.072439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.073052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.074940 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.075603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 791/1587] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.076544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.087209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f16a35-2840-4abc-b0a4-4c71e5a4d37b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.092765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40591ab-5e18-4e5f-993f-a32ccbb87357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.093419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40591ab-5e18-4e5f-993f-a32ccbb87357 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.093762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40591ab-5e18-4e5f-993f-a32ccbb87357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.094115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40591ab-5e18-4e5f-993f-a32ccbb87357 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.094792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40591ab-5e18-4e5f-993f-a32ccbb87357 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.095442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 797/1588] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/users => generated 297 bytes in 39 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:45.096551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40591ab-5e18-4e5f-993f-a32ccbb87357 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.099217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.106896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 792/1589] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.108165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df549e6c-47a4-4b96-a4c2-0e1417251957 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.108506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df549e6c-47a4-4b96-a4c2-0e1417251957 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.108701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc990763-56fd-4b2c-bd4a-19b9af32c16f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.108890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df549e6c-47a4-4b96-a4c2-0e1417251957 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.113538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.114116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.114395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.114664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.115313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.116918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.128372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 798/1590] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.132340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.137409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-df549e6c-47a4-4b96-a4c2-0e1417251957 None None] MFA Rules not processed for user `2bd4786e1b444c318436ead072ca2541`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:45.138600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-df549e6c-47a4-4b96-a4c2-0e1417251957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:45.142359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d746862-b887-4b70-9578-0e0ec6d481fc admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.147080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.147716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.148026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.148302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] PATH_INFO: `/v3/projects/477dd0575c8741b0815ddf5f22297db3/users/a39440ed016445a7bfc7ac7fba6661f2/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.163225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] RBAC: Authorizing `identity:create_grant(project_id=477dd0575c8741b0815ddf5f22297db3, user_id=a39440ed016445a7bfc7ac7fba6661f2, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.165963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.179261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 799/1591] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:45 2022] PUT /identity/v3/projects/477dd0575c8741b0815ddf5f22297db3/users/a39440ed016445a7bfc7ac7fba6661f2/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:45.180726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.207025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 793/1592] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:45.208036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df549e6c-47a4-4b96-a4c2-0e1417251957 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.218865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df549e6c-47a4-4b96-a4c2-0e1417251957 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.233419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:45.261334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea506037-e4ac-4a9c-8fa2-b5e275406faa admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.265952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.266617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.266895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.267165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] PATH_INFO: `/v3/projects/1377f60423c64a1786203deb55ae55ef/users/dd95ef263d15460f9cca5a7557cbee2e/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.282359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.282540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=1377f60423c64a1786203deb55ae55ef, user_id=dd95ef263d15460f9cca5a7557cbee2e, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.282963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.283242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.283509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.284175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.285182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.285756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.296982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 794/1593] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.297870 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.298985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 800/1594] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:45 2022] PUT /identity/v3/projects/1377f60423c64a1786203deb55ae55ef/users/dd95ef263d15460f9cca5a7557cbee2e/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:45.299832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.305042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:45.309523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3f3291-3c54-4a5f-acba-86198fdda08c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.364888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d19c080-007f-4379-a0c6-f500d40f21eb admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.368018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.368619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.368892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.369159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.369794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.369932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.370394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.370674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.370942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.371397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.371576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.373184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.381908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 795/1595] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.382681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.386918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 801/1596] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.386918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.392415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39fc2900-9eb5-4bc7-964f-0b87d3b10937 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.396501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b7cbf2-70e1-4b83-bf4f-c5ff66758a8d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.397073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5cc10c-a428-4eba-a687-22bdb9fcdcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.397755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5cc10c-a428-4eba-a687-22bdb9fcdcd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.398029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5cc10c-a428-4eba-a687-22bdb9fcdcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.398297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5cc10c-a428-4eba-a687-22bdb9fcdcd3 admin admin] PATH_INFO: `/v3/projects/7b8c9cc2c53d43a895e570024b80d98b/users/167d5891f699468b915a620c1db66339/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.401888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55dee2a-9008-40c9-b27c-e8ae5ec3f7c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.402598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dee2a-9008-40c9-b27c-e8ae5ec3f7c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.402884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dee2a-9008-40c9-b27c-e8ae5ec3f7c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.403164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dee2a-9008-40c9-b27c-e8ae5ec3f7c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.403840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55dee2a-9008-40c9-b27c-e8ae5ec3f7c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.405476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55dee2a-9008-40c9-b27c-e8ae5ec3f7c5 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.414383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5cc10c-a428-4eba-a687-22bdb9fcdcd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b8c9cc2c53d43a895e570024b80d98b, user_id=167d5891f699468b915a620c1db66339, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.415586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 802/1597] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.415586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0135a4-ec0c-47a4-b2e2-f11337792340 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.415586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0135a4-ec0c-47a4-b2e2-f11337792340 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.419008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5cc10c-a428-4eba-a687-22bdb9fcdcd3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.419122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0135a4-ec0c-47a4-b2e2-f11337792340 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.431019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 796/1598] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:45 2022] PUT /identity/v3/projects/7b8c9cc2c53d43a895e570024b80d98b/users/167d5891f699468b915a620c1db66339/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:45.432457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa4afc-7a45-4945-95ff-ddc191187481 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.432840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa4afc-7a45-4945-95ff-ddc191187481 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.433175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa4afc-7a45-4945-95ff-ddc191187481 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.434466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 797/1599] 10.222.0.42 () {54 vars in 962 bytes} [Fri Sep 16 01:53:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:53:45.435226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa4afc-7a45-4945-95ff-ddc191187481 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.448687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4b0135a4-ec0c-47a4-b2e2-f11337792340 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:45.449966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4b0135a4-ec0c-47a4-b2e2-f11337792340 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:45.501866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa4afc-7a45-4945-95ff-ddc191187481 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.506775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.507370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.507648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.507997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.508635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.510218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.512507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 803/1600] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:45.513711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356b6dc-1ddf-449e-8d56-dae2a7dbf028 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.514035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356b6dc-1ddf-449e-8d56-dae2a7dbf028 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.514320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f356b6dc-1ddf-449e-8d56-dae2a7dbf028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.523819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 798/1601] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.523819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.533228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cc8764-ed1c-497c-b18f-078c6d85a21b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.536649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f356b6dc-1ddf-449e-8d56-dae2a7dbf028 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:45.537892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f356b6dc-1ddf-449e-8d56-dae2a7dbf028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:45.538091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fae484-8486-476f-a183-c9b2d76743ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.538808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fae484-8486-476f-a183-c9b2d76743ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.539827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fae484-8486-476f-a183-c9b2d76743ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.540256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fae484-8486-476f-a183-c9b2d76743ae admin admin] PATH_INFO: `/v3/projects/eddbb633debb4badb74f142d92baed98/users/6fc9485505c443acb946ec66e0018f7b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.547323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 804/1602] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:45.548232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356b6dc-1ddf-449e-8d56-dae2a7dbf028 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.558598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f356b6dc-1ddf-449e-8d56-dae2a7dbf028 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.560057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fae484-8486-476f-a183-c9b2d76743ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=eddbb633debb4badb74f142d92baed98, user_id=6fc9485505c443acb946ec66e0018f7b, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.562936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fae484-8486-476f-a183-c9b2d76743ae admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.563499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.564188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.564467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.564731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.565359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.566117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:45.567028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.579596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 805/1603] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:45.580272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.583085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 799/1604] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:45 2022] PUT /identity/v3/projects/eddbb633debb4badb74f142d92baed98/users/6fc9485505c443acb946ec66e0018f7b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:45.584450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bace1a-88eb-43e3-9205-fccad32a4673 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.584729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bace1a-88eb-43e3-9205-fccad32a4673 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.584994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15bace1a-88eb-43e3-9205-fccad32a4673 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.590819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766079e9-8ff4-4081-962d-db204280799e admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.613578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-15bace1a-88eb-43e3-9205-fccad32a4673 None None] MFA Rules not processed for user `167d5891f699468b915a620c1db66339`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:45.614796 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-15bace1a-88eb-43e3-9205-fccad32a4673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:45.651080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.651795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.652108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.652380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] PATH_INFO: `/v3/projects/f6ad8527ba8a453bad0232fdcb88e5ca/users/d2165238c4614ec2894b763a60a53e31/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.667139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6ad8527ba8a453bad0232fdcb88e5ca, user_id=d2165238c4614ec2894b763a60a53e31, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.669892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.673489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:45.680511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 800/1605] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:45.681931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d899e4-3aa3-48b6-a1a3-d01afac749a7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.682158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d899e4-3aa3-48b6-a1a3-d01afac749a7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.682435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d899e4-3aa3-48b6-a1a3-d01afac749a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.685388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 806/1606] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:45 2022] PUT /identity/v3/projects/f6ad8527ba8a453bad0232fdcb88e5ca/users/d2165238c4614ec2894b763a60a53e31/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:45.689722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.721461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-38d899e4-3aa3-48b6-a1a3-d01afac749a7 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:45.722664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-38d899e4-3aa3-48b6-a1a3-d01afac749a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:45.748933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ae84c0-c10e-4770-ae53-d9626b9a81d4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.752653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.753255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.753536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.753839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.789610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 801/1607] 10.222.0.42 () {58 vars in 1059 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:45.790530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d899e4-3aa3-48b6-a1a3-d01afac749a7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.819679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.821442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.830837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 807/1608] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:45.831645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.832943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:45.844089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f923ccd-f878-4e05-91ac-b714e833336d service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.848086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.848691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.848971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.849242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.858812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d899e4-3aa3-48b6-a1a3-d01afac749a7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.859240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.860708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.863835 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe3ca09-edc3-4f89-a717-88d1638dbf5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.864582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3ca09-edc3-4f89-a717-88d1638dbf5e admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.864923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3ca09-edc3-4f89-a717-88d1638dbf5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.865199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe3ca09-edc3-4f89-a717-88d1638dbf5e admin admin] PATH_INFO: `/v3/projects/45db59daaa1c42e689bb38c8e6ce6820/users/cc17bab3e54b42a7b367919371a07c8f/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.871772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 808/1609] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:45.872614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:45.876510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:45.885244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07936504-98b6-42ec-8f09-e9187135f7f9 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:45.886520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe3ca09-edc3-4f89-a717-88d1638dbf5e admin admin] RBAC: Authorizing `identity:create_grant(project_id=45db59daaa1c42e689bb38c8e6ce6820, user_id=cc17bab3e54b42a7b367919371a07c8f, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.890349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe3ca09-edc3-4f89-a717-88d1638dbf5e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.891485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:45.892542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.893079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.893502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] PATH_INFO: `/v3/projects/57aaf234892543698d91303961f14abd/users/7eb6ab0644bf4fa59d7bba232d6ccb3c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.926648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 802/1610] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:45 2022] PUT /identity/v3/projects/45db59daaa1c42e689bb38c8e6ce6820/users/cc17bab3e54b42a7b367919371a07c8f/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:45.926648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897cf82a-b090-4dad-ba7c-b333b21cf651 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:45.926648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897cf82a-b090-4dad-ba7c-b333b21cf651 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:45.926648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897cf82a-b090-4dad-ba7c-b333b21cf651 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:45.934443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57aaf234892543698d91303961f14abd, user_id=7eb6ab0644bf4fa59d7bba232d6ccb3c, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:45.942351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-897cf82a-b090-4dad-ba7c-b333b21cf651 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:45.943827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-897cf82a-b090-4dad-ba7c-b333b21cf651 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:45.954203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:45.996114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 809/1611] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:45 2022] PUT /identity/v3/projects/57aaf234892543698d91303961f14abd/users/7eb6ab0644bf4fa59d7bba232d6ccb3c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:45.996114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.033164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 803/1612] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:46.037802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6fb869-a86c-410a-8bf5-ce0a37e39c44 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.038095 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6fb869-a86c-410a-8bf5-ce0a37e39c44 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.038419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6fb869-a86c-410a-8bf5-ce0a37e39c44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.068229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2c6fb869-a86c-410a-8bf5-ce0a37e39c44 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:46.069453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2c6fb869-a86c-410a-8bf5-ce0a37e39c44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:46.104098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffcfa95-031e-4b8e-8701-70af4f19cdb3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.111765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.112794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.113270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.113807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.114560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.117740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.135361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:46.156814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 810/1613] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:45 2022] POST /identity/v3/projects => generated 363 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:46.157817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.169361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 804/1614] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:46.172307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93aaeb75-f3ec-4822-94bd-b538ff72ef9b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.172595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93aaeb75-f3ec-4822-94bd-b538ff72ef9b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.173019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93aaeb75-f3ec-4822-94bd-b538ff72ef9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.208912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-93aaeb75-f3ec-4822-94bd-b538ff72ef9b None None] MFA Rules not processed for user `cc17bab3e54b42a7b367919371a07c8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:46.210148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-93aaeb75-f3ec-4822-94bd-b538ff72ef9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:46.298773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8f32c0-fd84-44e7-9598-ec3d0e019c10 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.310622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 805/1615] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:46 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:46.310622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740a6e5-19cc-492f-95cc-bd37f2ff8b31 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.310622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740a6e5-19cc-492f-95cc-bd37f2ff8b31 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.310622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b740a6e5-19cc-492f-95cc-bd37f2ff8b31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.329460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.330096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.330380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.330655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.337293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b740a6e5-19cc-492f-95cc-bd37f2ff8b31 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:46.338952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b740a6e5-19cc-492f-95cc-bd37f2ff8b31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:46.368094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:46.423630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.426027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.441825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 811/1616] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:46.442904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.482111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 806/1617] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:46.483123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740a6e5-19cc-492f-95cc-bd37f2ff8b31 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.504620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b740a6e5-19cc-492f-95cc-bd37f2ff8b31 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.516685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.517936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.518280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.518624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.519511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.521898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.536472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb208c6-35e8-4d4d-897e-610e17c4dd89 service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.541165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.542031 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.543898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.544209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.558426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 807/1618] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:46 2022] POST /identity/v3/users => generated 288 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:46.558426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.568076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.571989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a044680b-767e-4c7c-8e34-4f8120c5f8aa admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.573059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.577462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.579050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.579762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.580952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.589011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 812/1619] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:46.590195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.601775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.603364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.603890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2c1f46-9974-4454-9572-69ffc8628d14 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.609205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.610098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.610578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.611440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.612426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.614560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.627378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 808/1620] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:46.633691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.651082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58daac9-45cf-41a7-adbf-3b973f848e92 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.660338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.661080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.662781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.663159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.666574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 813/1621] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:46 2022] POST /identity/v3/projects => generated 357 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:46.667847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.686290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.687979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:46.691461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.718728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 809/1622] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:46.721293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.746750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2002996-eb6e-4dc5-8882-d34fda3b6649 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.786881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505d3e52-6226-4c29-86bb-05f931943e2f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.794245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ad4710-3c5f-4062-9e5a-dc753628c180 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.795114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad4710-3c5f-4062-9e5a-dc753628c180 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.795550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad4710-3c5f-4062-9e5a-dc753628c180 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.796015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ad4710-3c5f-4062-9e5a-dc753628c180 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.800989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ad4710-3c5f-4062-9e5a-dc753628c180 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.802739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ad4710-3c5f-4062-9e5a-dc753628c180 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.818885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 814/1623] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:46 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:46.820600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee644cb-bbd0-4e19-a9a9-1a5f8dfde3e9 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.820892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee644cb-bbd0-4e19-a9a9-1a5f8dfde3e9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.821366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee644cb-bbd0-4e19-a9a9-1a5f8dfde3e9 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.832331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 815/1624] 10.222.0.42 () {54 vars in 912 bytes} [Fri Sep 16 01:53:46 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:53:46.834361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee644cb-bbd0-4e19-a9a9-1a5f8dfde3e9 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:46.856954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee644cb-bbd0-4e19-a9a9-1a5f8dfde3e9 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:46.863822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e4b06b-a19d-4770-b40b-cfdc74d4b02f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.864786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e4b06b-a19d-4770-b40b-cfdc74d4b02f admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.865455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e4b06b-a19d-4770-b40b-cfdc74d4b02f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.865455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e4b06b-a19d-4770-b40b-cfdc74d4b02f admin admin] PATH_INFO: `/v3/projects/e1de48fb8c9542e399b86c1d76151b00/users/3ab0fcd27cbb473b87793afe15784e48/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.892695 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e4b06b-a19d-4770-b40b-cfdc74d4b02f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1de48fb8c9542e399b86c1d76151b00, user_id=3ab0fcd27cbb473b87793afe15784e48, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.895886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e4b06b-a19d-4770-b40b-cfdc74d4b02f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.917230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 816/1625] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:46 2022] PUT /identity/v3/projects/e1de48fb8c9542e399b86c1d76151b00/users/3ab0fcd27cbb473b87793afe15784e48/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:46.918834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90557f7-af28-48c1-8227-f890d32157c8 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.919123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90557f7-af28-48c1-8227-f890d32157c8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.919993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90557f7-af28-48c1-8227-f890d32157c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.925688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:46.926579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:46.927010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:46.927490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:46.928360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:46.930453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:46.948137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d90557f7-af28-48c1-8227-f890d32157c8 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:46.949949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d90557f7-af28-48c1-8227-f890d32157c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:46.985126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 810/1626] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:46 2022] POST /identity/v3/users => generated 285 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:46.986278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.001825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b074a71-7b40-4ff5-af33-6d4dc56851f1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.049260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 817/1627] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:53:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:47.056492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90557f7-af28-48c1-8227-f890d32157c8 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.135578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.137173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.137506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.137783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.139710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.144424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.157846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:47.174566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 811/1628] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:46 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:47.174799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90557f7-af28-48c1-8227-f890d32157c8 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.174905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.184026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.186491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.186775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.187101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.188581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.194269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.219477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 818/1629] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:47 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:47.221716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.252857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999a6844-6dcc-4187-8c09-ca4e22888962 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.260257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.261140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.261712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.262092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] PATH_INFO: `/v3/projects/e1de48fb8c9542e399b86c1d76151b00/users/3ab0fcd27cbb473b87793afe15784e48/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.264422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1de48fb8c9542e399b86c1d76151b00, user_id=3ab0fcd27cbb473b87793afe15784e48, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.267673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.276073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:47.292067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 819/1630] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:47 2022] PUT /identity/v3/projects/e1de48fb8c9542e399b86c1d76151b00/users/3ab0fcd27cbb473b87793afe15784e48/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:47.292067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.377150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d5987-e8d5-4e40-be5e-a97c18963228 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.392927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6e040a-0fe5-40ef-9bfa-877b052922ab admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.412084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825563cc-8bb5-423f-a20e-8e469db4cee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.414162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825563cc-8bb5-423f-a20e-8e469db4cee8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.414162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825563cc-8bb5-423f-a20e-8e469db4cee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.414162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825563cc-8bb5-423f-a20e-8e469db4cee8 admin admin] PATH_INFO: `/v3/projects/89bbdf6516124d80877954950aeeab29/users/9640ac96ecee44f88793bcc79c5c630b/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.471724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825563cc-8bb5-423f-a20e-8e469db4cee8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89bbdf6516124d80877954950aeeab29, user_id=9640ac96ecee44f88793bcc79c5c630b, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.478072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825563cc-8bb5-423f-a20e-8e469db4cee8 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.507442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.508806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.509260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.509895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.515465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 820/1631] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:47 2022] PUT /identity/v3/projects/89bbdf6516124d80877954950aeeab29/users/9640ac96ecee44f88793bcc79c5c630b/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:47.516170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24220ede-62a0-4d3b-812c-1610666471e7 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.516452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24220ede-62a0-4d3b-812c-1610666471e7 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.516736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24220ede-62a0-4d3b-812c-1610666471e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.583278 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-24220ede-62a0-4d3b-812c-1610666471e7 None None] MFA Rules not processed for user `3ab0fcd27cbb473b87793afe15784e48`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:47.587264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-24220ede-62a0-4d3b-812c-1610666471e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:47.651213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.653931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.669627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 812/1632] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:47.671227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.687254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9193fc4f-bc64-4371-acc6-763e5466fec2 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.743921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 821/1633] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:47 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:47.758778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24220ede-62a0-4d3b-812c-1610666471e7 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.780418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:47.796739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.800791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.800791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.800791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.800791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.809030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.843539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 813/1634] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:47 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:47.845708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:47.869847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e75049a-2585-43eb-8a17-73c558a3754b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.880750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd81e6a9-c79f-497f-8a44-3baf07f004b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.881422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81e6a9-c79f-497f-8a44-3baf07f004b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.881700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81e6a9-c79f-497f-8a44-3baf07f004b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.881972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd81e6a9-c79f-497f-8a44-3baf07f004b0 admin admin] PATH_INFO: `/v3/projects/89bbdf6516124d80877954950aeeab29/users/9640ac96ecee44f88793bcc79c5c630b/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.890981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd81e6a9-c79f-497f-8a44-3baf07f004b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89bbdf6516124d80877954950aeeab29, user_id=9640ac96ecee44f88793bcc79c5c630b, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.898462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24220ede-62a0-4d3b-812c-1610666471e7 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:47.899577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd81e6a9-c79f-497f-8a44-3baf07f004b0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.910725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:47.911349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.911624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.912302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.926046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 814/1635] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:47 2022] PUT /identity/v3/projects/89bbdf6516124d80877954950aeeab29/users/9640ac96ecee44f88793bcc79c5c630b/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:47.936959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3021a556-0cda-4262-a180-64fc67d25944 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:47.937248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3021a556-0cda-4262-a180-64fc67d25944 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:47.937520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3021a556-0cda-4262-a180-64fc67d25944 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:47.944542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:47.947739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:47.974372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 822/1636] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:47 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:53:48.002259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3021a556-0cda-4262-a180-64fc67d25944 None None] MFA Rules not processed for user `9640ac96ecee44f88793bcc79c5c630b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:48.004171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3021a556-0cda-4262-a180-64fc67d25944 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:48.130960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 815/1637] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:47 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:48.169019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:48.185395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c425518d-67df-4bc0-9d67-c82c22dfb1c5 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:48.197977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:48.300858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:48.307256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:48.308583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:48.309848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:48.326455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:48.328787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:48.362953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 823/1638] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:48 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:48.461364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3021a556-0cda-4262-a180-64fc67d25944 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:48.482748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3021a556-0cda-4262-a180-64fc67d25944 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:48.490153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e55cc-d4a3-47e6-9a18-c6654c13acda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:48.492476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e55cc-d4a3-47e6-9a18-c6654c13acda service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:48.493045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e55cc-d4a3-47e6-9a18-c6654c13acda service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:48.493518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e55cc-d4a3-47e6-9a18-c6654c13acda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:48.604507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e55cc-d4a3-47e6-9a18-c6654c13acda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:48.606022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e55cc-d4a3-47e6-9a18-c6654c13acda service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:48.617138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 816/1639] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:48 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:48.795442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:48.812154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a08e3ff-4826-48a1-aff3-5ef4052a0c36 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:48.824139 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c663b104-facf-474c-9c40-deedf42fb0c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:48.825381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663b104-facf-474c-9c40-deedf42fb0c6 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:48.828790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663b104-facf-474c-9c40-deedf42fb0c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:48.830200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c663b104-facf-474c-9c40-deedf42fb0c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:49.043435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c663b104-facf-474c-9c40-deedf42fb0c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:49.043435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c663b104-facf-474c-9c40-deedf42fb0c6 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:49.066136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 824/1640] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:48 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:49.356225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b7b6f-441b-42a6-bf8e-b1e3c7a4cfb2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:49.356803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b7b6f-441b-42a6-bf8e-b1e3c7a4cfb2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:49.356920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197b7b6f-441b-42a6-bf8e-b1e3c7a4cfb2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:49.358579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 817/1641] 10.222.0.42 () {54 vars in 962 bytes} [Fri Sep 16 01:53:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:53:49.373332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9180b7-244c-4431-b24f-13c4c5714d73 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:49.373934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9180b7-244c-4431-b24f-13c4c5714d73 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:49.374397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9180b7-244c-4431-b24f-13c4c5714d73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:49.501880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1b9180b7-244c-4431-b24f-13c4c5714d73 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:49.501880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1b9180b7-244c-4431-b24f-13c4c5714d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:49.689539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 825/1642] 10.222.0.42 () {58 vars in 1059 bytes} [Fri Sep 16 01:53:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 319 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:49.697389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26427cd0-5cd4-42cb-a675-279934292288 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:49.697594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26427cd0-5cd4-42cb-a675-279934292288 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:49.697850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26427cd0-5cd4-42cb-a675-279934292288 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:49.699297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 818/1643] 10.222.0.42 () {54 vars in 912 bytes} [Fri Sep 16 01:53:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:53:49.706761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a40b799-f1aa-40d6-b808-e36093bf564d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:49.707214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a40b799-f1aa-40d6-b808-e36093bf564d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:49.707657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a40b799-f1aa-40d6-b808-e36093bf564d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:49.774217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4a40b799-f1aa-40d6-b808-e36093bf564d None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:49.775675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4a40b799-f1aa-40d6-b808-e36093bf564d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:49.831713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 826/1644] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:53:49 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:49.850812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26427cd0-5cd4-42cb-a675-279934292288 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:49.867750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26427cd0-5cd4-42cb-a675-279934292288 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:49.884595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:49.885284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:49.885569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:49.885845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:49.910026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:49.911961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:49.929529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 819/1645] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:50.580570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102f1f91-018e-4ec0-9bae-1e081aa6ca2c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:50.581432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102f1f91-018e-4ec0-9bae-1e081aa6ca2c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:50.582088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102f1f91-018e-4ec0-9bae-1e081aa6ca2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:50.659992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-102f1f91-018e-4ec0-9bae-1e081aa6ca2c None None] MFA Rules not processed for user `d2165238c4614ec2894b763a60a53e31`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:50.673311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-102f1f91-018e-4ec0-9bae-1e081aa6ca2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:50.731349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:50.831086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 827/1646] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:50 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:50.835583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:51.060123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:51.090922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a5f94-3c04-486f-aa9e-088e2ecd5dd3 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:51.107564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:51.108546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:51.109093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:51.109505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:51.110619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:51.114790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:51.196566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 820/1647] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:53:50 2022] POST /identity/v3/projects => generated 405 bytes in 361 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:51.201724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102f1f91-018e-4ec0-9bae-1e081aa6ca2c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:51.221856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102f1f91-018e-4ec0-9bae-1e081aa6ca2c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:51.384342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:51.385206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:51.385527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:51.385919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:51.386726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:51.391424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:51.457874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:51.459610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 828/1648] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:53:51 2022] POST /identity/v3/users => generated 309 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:53:51.475927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0395cd-cb4a-4567-837c-42a361e80818 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:51.482061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:51.482983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:51.483770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:51.484413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:51.485501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:51.490844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:51.511656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 821/1649] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:51 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:51.511863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:51.531750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ea5aa-cfad-4a33-9df5-d3b21bac2c54 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:51.543487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:51.547015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:51.547504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:51.548044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] PATH_INFO: `/v3/projects/01306860aea24260922ede8ae7973aa9/users/98bacdee0de94c148df2b02c663f8933/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:51.578404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01306860aea24260922ede8ae7973aa9, user_id=98bacdee0de94c148df2b02c663f8933, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:51.583013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:51.609339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 829/1650] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:51 2022] PUT /identity/v3/projects/01306860aea24260922ede8ae7973aa9/users/98bacdee0de94c148df2b02c663f8933/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:53:51.613540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:51.626936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1e5c78-a519-4baf-b091-3a5c796f79af admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:51.733379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d23210-e452-4f5d-979c-b04abec599e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:51.734078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d23210-e452-4f5d-979c-b04abec599e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:51.734424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d23210-e452-4f5d-979c-b04abec599e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:51.734850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d23210-e452-4f5d-979c-b04abec599e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:51.735915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d23210-e452-4f5d-979c-b04abec599e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:51.737918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d23210-e452-4f5d-979c-b04abec599e2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:51.770663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 822/1651] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:53:51 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:53:51.775796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:51.814846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae8bfa-a156-40cc-b036-eed5c8d46e23 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:51.829891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8609f5-d3e6-489f-b206-5b100dc607c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:51.830878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8609f5-d3e6-489f-b206-5b100dc607c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:51.831294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8609f5-d3e6-489f-b206-5b100dc607c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:51.831689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8609f5-d3e6-489f-b206-5b100dc607c8 admin admin] PATH_INFO: `/v3/projects/01306860aea24260922ede8ae7973aa9/users/98bacdee0de94c148df2b02c663f8933/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:51.838896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8609f5-d3e6-489f-b206-5b100dc607c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01306860aea24260922ede8ae7973aa9, user_id=98bacdee0de94c148df2b02c663f8933, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:51.842746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8609f5-d3e6-489f-b206-5b100dc607c8 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:51.867742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 830/1652] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:53:51 2022] PUT /identity/v3/projects/01306860aea24260922ede8ae7973aa9/users/98bacdee0de94c148df2b02c663f8933/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:53:54.618213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde1090-4202-4121-b051-e57382af9562 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:54.619056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde1090-4202-4121-b051-e57382af9562 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:54.619474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fde1090-4202-4121-b051-e57382af9562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:54.696052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0fde1090-4202-4121-b051-e57382af9562 None None] MFA Rules not processed for user `98bacdee0de94c148df2b02c663f8933`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:54.697299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0fde1090-4202-4121-b051-e57382af9562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:54.872300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 823/1653] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:54 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:57.861104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566ac0ab-e953-4ce6-b252-0edbef7505c5 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:57.861692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566ac0ab-e953-4ce6-b252-0edbef7505c5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:57.861820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566ac0ab-e953-4ce6-b252-0edbef7505c5 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:57.863782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 831/1654] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:53:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:53:57.887054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b02df58-2c90-48b4-9e98-3e022cc032e0 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:57.887560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b02df58-2c90-48b4-9e98-3e022cc032e0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:57.887634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b02df58-2c90-48b4-9e98-3e022cc032e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:57.934735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3b02df58-2c90-48b4-9e98-3e022cc032e0 None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:57.935987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3b02df58-2c90-48b4-9e98-3e022cc032e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:58.068484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:58.120311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 824/1655] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:53:57 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:58.144573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566ac0ab-e953-4ce6-b252-0edbef7505c5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:58.162756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566ac0ab-e953-4ce6-b252-0edbef7505c5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:58.177149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b56a0c-de64-4ca8-a948-d89634975453 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:58.177968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b56a0c-de64-4ca8-a948-d89634975453 service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:58.183178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b56a0c-de64-4ca8-a948-d89634975453 service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:58.183674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b56a0c-de64-4ca8-a948-d89634975453 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:58.209360 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b56a0c-de64-4ca8-a948-d89634975453 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:58.212128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b56a0c-de64-4ca8-a948-d89634975453 service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:58.260114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 832/1656] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:58 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:58.829389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da985816-14fb-4a38-98db-32fd6a2f0902 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:58.830419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da985816-14fb-4a38-98db-32fd6a2f0902 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:58.830847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da985816-14fb-4a38-98db-32fd6a2f0902 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:58.921001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-da985816-14fb-4a38-98db-32fd6a2f0902 None None] MFA Rules not processed for user `a39440ed016445a7bfc7ac7fba6661f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:58.925086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-da985816-14fb-4a38-98db-32fd6a2f0902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:59.067659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 825/1657] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:59.081656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d3120a-9b87-4767-ac2c-90ae4d430620 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:59.082129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d3120a-9b87-4767-ac2c-90ae4d430620 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:59.082350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d3120a-9b87-4767-ac2c-90ae4d430620 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:59.147914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b3d3120a-9b87-4767-ac2c-90ae4d430620 None None] MFA Rules not processed for user `a39440ed016445a7bfc7ac7fba6661f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:53:59.150015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b3d3120a-9b87-4767-ac2c-90ae4d430620 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:53:59.160104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 833/1658] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:53:59 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:53:59.174812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da985816-14fb-4a38-98db-32fd6a2f0902 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:59.353896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da985816-14fb-4a38-98db-32fd6a2f0902 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:59.361121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46459a3e-b988-4bdc-871f-1856c3677d05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:59.361912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46459a3e-b988-4bdc-871f-1856c3677d05 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:59.362193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46459a3e-b988-4bdc-871f-1856c3677d05 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:59.362684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46459a3e-b988-4bdc-871f-1856c3677d05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:59.383572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46459a3e-b988-4bdc-871f-1856c3677d05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:59.385701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46459a3e-b988-4bdc-871f-1856c3677d05 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:53:59.404867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 826/1659] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:53:59 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:53:59.696104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d3120a-9b87-4767-ac2c-90ae4d430620 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:53:59.899830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:53:59.916384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d3120a-9b87-4767-ac2c-90ae4d430620 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:53:59.940679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:53:59.941336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:53:59.941619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:53:59.942022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] PATH_INFO: `/v3/users/d2165238c4614ec2894b763a60a53e31` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:53:59.944249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2165238c4614ec2894b763a60a53e31)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:53:59.945942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: WARNING py.warnings [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:53:59.945942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: warnings.warn(msg) Sep 16 01:53:59.945942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: Sep 16 01:53:59.948008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:00.073903 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:00.074372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:00.074694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:00.159652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2165238c4614ec2894b763a60a53e31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:00.160875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2165238c4614ec2894b763a60a53e31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:00.163877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2165238c4614ec2894b763a60a53e31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:00.170568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] MFA Rules not processed for user `dd95ef263d15460f9cca5a7557cbee2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:00.171800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:00.198955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2165238c4614ec2894b763a60a53e31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:00.270311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-178023ba-2770-4e15-af40-25cdc9e3aef6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2165238c4614ec2894b763a60a53e31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:00.299663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 834/1660] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:53:59 2022] DELETE /identity/v3/users/d2165238c4614ec2894b763a60a53e31 => generated 0 bytes in 605 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:00.300798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:00.333884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 827/1661] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:00 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:00.350162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:00.350517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66974f-56c2-4b02-a395-2ab7b89d8a37 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:00.350995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66974f-56c2-4b02-a395-2ab7b89d8a37 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:00.351254 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66974f-56c2-4b02-a395-2ab7b89d8a37 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:00.351684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:00.353551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 835/1662] 10.222.0.42 () {54 vars in 912 bytes} [Fri Sep 16 01:54:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:54:00.359039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:00.360997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:00.361491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:00.375470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a2a185-d252-44ef-b3b6-793f4373ac37 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:00.486179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] MFA Rules not processed for user `6a343a260fa2415e84caf959933a6c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:00.488564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:00.597698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa11890-a1dd-496b-b627-dc5c4c9af1a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:00.599275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa11890-a1dd-496b-b627-dc5c4c9af1a8 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:00.602511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa11890-a1dd-496b-b627-dc5c4c9af1a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:00.603412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa11890-a1dd-496b-b627-dc5c4c9af1a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:00.618664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 836/1663] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:54:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:00.647526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:00.649040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:00.673859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93610f85-39ef-43e3-be59-a93740cba8ec None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:00.684378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:00.685754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:00.686276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:00.686748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:00.708144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa11890-a1dd-496b-b627-dc5c4c9af1a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:00.709767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa11890-a1dd-496b-b627-dc5c4c9af1a8 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:00.748766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 828/1664] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:00 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:00.763461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:00.765348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:00.790213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 837/1665] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:01.178598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c72608-f1bd-44c6-9fb3-fa7d62157a14 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:01.179560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c72608-f1bd-44c6-9fb3-fa7d62157a14 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:01.180622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c72608-f1bd-44c6-9fb3-fa7d62157a14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:01.283982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f4c72608-f1bd-44c6-9fb3-fa7d62157a14 None None] MFA Rules not processed for user `6fc9485505c443acb946ec66e0018f7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:01.298119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f4c72608-f1bd-44c6-9fb3-fa7d62157a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:01.514497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 829/1666] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:01 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 337 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:01.515374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:01.516971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:01.521906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c72608-f1bd-44c6-9fb3-fa7d62157a14 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:01.543257 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c72608-f1bd-44c6-9fb3-fa7d62157a14 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:01.580983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:01.581663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:01.581894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:01.582803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:01.617624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:01.621047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:01.638324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:01.653135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 830/1667] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:01 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:01.703433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd77aef7-c241-4113-b216-d01f4ad69426 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:01.714036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:01.714950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:01.719744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:01.720225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] PATH_INFO: `/v3/projects/f6ad8527ba8a453bad0232fdcb88e5ca` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:01.722778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6ad8527ba8a453bad0232fdcb88e5ca)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:01.724947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:01.731342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:01.886295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6ad8527ba8a453bad0232fdcb88e5ca', 'request_id': 'req-34749d40-a17a-43e8-bc85-253bbbb4c07d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:01.896637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6ad8527ba8a453bad0232fdcb88e5ca', 'request_id': 'req-34749d40-a17a-43e8-bc85-253bbbb4c07d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:01.911869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6ad8527ba8a453bad0232fdcb88e5ca', 'request_id': 'req-34749d40-a17a-43e8-bc85-253bbbb4c07d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:01.938610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] The token cache is being invalidate because project f6ad8527ba8a453bad0232fdcb88e5ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:01.939157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:01.940244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:01.943317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 838/1668] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:01 2022] DELETE /identity/v3/projects/f6ad8527ba8a453bad0232fdcb88e5ca => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:01.948676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:01.968971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5746ef17-d1c5-4577-a852-e3aeb8cd7ddb service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:02.253178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:02.253838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:02.254772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:02.254772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] PATH_INFO: `/v3/users/98bacdee0de94c148df2b02c663f8933` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:02.263865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] RBAC: Authorizing `identity:delete_user(user_id=98bacdee0de94c148df2b02c663f8933)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:02.269329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:02.359868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:02.361588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:02.432361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98bacdee0de94c148df2b02c663f8933'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:02.437312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98bacdee0de94c148df2b02c663f8933'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:02.468362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98bacdee0de94c148df2b02c663f8933'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:02.472475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98bacdee0de94c148df2b02c663f8933'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:02.507511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98bacdee0de94c148df2b02c663f8933'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:02.546676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 831/1669] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:01 2022] DELETE /identity/v3/users/98bacdee0de94c148df2b02c663f8933 => generated 0 bytes in 598 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:02.570185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34749d40-a17a-43e8-bc85-253bbbb4c07d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:02.773756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f901ef-beea-42f9-ac0d-486faf9fa3e2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:02.774715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f901ef-beea-42f9-ac0d-486faf9fa3e2 service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:02.775268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f901ef-beea-42f9-ac0d-486faf9fa3e2 service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:02.775825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f901ef-beea-42f9-ac0d-486faf9fa3e2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:02.778402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a7f901ef-beea-42f9-ac0d-486faf9fa3e2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:02.794315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f901ef-beea-42f9-ac0d-486faf9fa3e2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:02.796422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f901ef-beea-42f9-ac0d-486faf9fa3e2 service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:02.811977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 839/1670] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:03.052505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:03.055662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:03.274867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a4b6ea-772d-4b66-81e2-9ed58de1789a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:03.287374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:03.295605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:03.295901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:03.296326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] PATH_INFO: `/v3/projects/01306860aea24260922ede8ae7973aa9` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:03.297695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01306860aea24260922ede8ae7973aa9)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:03.298746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: WARNING py.warnings [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 16 01:54:03.298746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: warnings.warn(msg) Sep 16 01:54:03.298746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: Sep 16 01:54:03.299975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:03.365603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:03.443541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01306860aea24260922ede8ae7973aa9', 'request_id': 'req-b4735c39-8164-4b7c-b280-ce37b6d710c7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:03.486538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01306860aea24260922ede8ae7973aa9', 'request_id': 'req-b4735c39-8164-4b7c-b280-ce37b6d710c7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:03.504904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01306860aea24260922ede8ae7973aa9', 'request_id': 'req-b4735c39-8164-4b7c-b280-ce37b6d710c7'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:03.537189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] The token cache is being invalidate because project 01306860aea24260922ede8ae7973aa9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:03.537508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:03.541980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b4735c39-8164-4b7c-b280-ce37b6d710c7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:03.545836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 832/1671] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:03 2022] DELETE /identity/v3/projects/01306860aea24260922ede8ae7973aa9 => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:03.550813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785e21b3-3e31-4615-a422-dc7c42197aff None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:03.551105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785e21b3-3e31-4615-a422-dc7c42197aff None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:03.551575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785e21b3-3e31-4615-a422-dc7c42197aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:03.615388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-785e21b3-3e31-4615-a422-dc7c42197aff None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:03.617934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-785e21b3-3e31-4615-a422-dc7c42197aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:03.661561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:03.781526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35536d98-14fc-403e-ac36-1722029b3866 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:03.782230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 840/1672] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:03.784450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35536d98-14fc-403e-ac36-1722029b3866 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:03.791115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35536d98-14fc-403e-ac36-1722029b3866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:03.851565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-35536d98-14fc-403e-ac36-1722029b3866 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:03.857479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-35536d98-14fc-403e-ac36-1722029b3866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:03.881729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 833/1673] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:03.886657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785e21b3-3e31-4615-a422-dc7c42197aff None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:03.899584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785e21b3-3e31-4615-a422-dc7c42197aff None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:03.923102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:03.925265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:03.926393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:03.927653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:03.928607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:03.932624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:03.940220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:03.962097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 841/1674] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:03.971834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35536d98-14fc-403e-ac36-1722029b3866 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:04.012230 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35536d98-14fc-403e-ac36-1722029b3866 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:04.018922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:04.019750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:04.020148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:04.020553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:04.021365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:04.024689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:04.059488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 834/1675] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:03 2022] POST /identity/v3/projects => generated 371 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:04.064128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:04.077924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57acef8-c52c-4632-9a70-361ad6654a89 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:04.191177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:04.193551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:04.193551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:04.193551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:04.193551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:04.196739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:04.233706 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 842/1676] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:04 2022] POST /identity/v3/users => generated 286 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:04.235025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:04.253014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7c155e-c9b2-4550-bcb9-25681e79324c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:04.266551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:04.267343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:04.267679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:04.268111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:04.269039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:04.271687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:04.284337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 835/1677] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:04 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:04.289044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:04.307282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6794e337-f07b-4e3a-8753-a8ef74e57924 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:04.314672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a28b419-1698-4274-8b9b-8213412e6d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:04.315344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28b419-1698-4274-8b9b-8213412e6d1b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:04.315664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28b419-1698-4274-8b9b-8213412e6d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:04.315961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a28b419-1698-4274-8b9b-8213412e6d1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:04.316785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a28b419-1698-4274-8b9b-8213412e6d1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:04.318486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a28b419-1698-4274-8b9b-8213412e6d1b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:04.357965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 843/1678] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:04 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:04.365563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:04.381391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f10b4de-4640-4604-bfb9-87956ae32efd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:04.392078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:04.393276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:04.393568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:04.393843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] PATH_INFO: `/v3/projects/2006fddfe87b49378370b389b68a3695/users/d407a4aee9b349518e71104f03549900/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:04.435068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2006fddfe87b49378370b389b68a3695, user_id=d407a4aee9b349518e71104f03549900, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:04.441673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:04.505984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a42ff6-773e-400a-bbd6-75022df98f3c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:04.506500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a42ff6-773e-400a-bbd6-75022df98f3c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:04.507474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a42ff6-773e-400a-bbd6-75022df98f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:04.507769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 836/1679] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:04 2022] PUT /identity/v3/projects/2006fddfe87b49378370b389b68a3695/users/d407a4aee9b349518e71104f03549900/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:04.613585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-67a42ff6-773e-400a-bbd6-75022df98f3c None None] MFA Rules not processed for user `d407a4aee9b349518e71104f03549900`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:04.616950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-67a42ff6-773e-400a-bbd6-75022df98f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:04.795708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:04.810569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 844/1680] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:04 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:05.066694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f16b68a-e97e-4c8e-a091-a24fc0902e9c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:05.088645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:05.089445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:05.089935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:05.090827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:05.091821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:05.103655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:05.188012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 837/1681] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:04 2022] POST /identity/v3/projects => generated 373 bytes in 393 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:05.191817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a42ff6-773e-400a-bbd6-75022df98f3c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:05.223971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a42ff6-773e-400a-bbd6-75022df98f3c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:05.272361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:05.394086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:05.394934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:05.395218 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:05.396217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:05.399607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:05.405017 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:05.455870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 845/1682] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:05 2022] POST /identity/v3/users => generated 293 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:05.456948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:05.476392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80464502-0443-428e-af36-68b5afeb776d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:05.486773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:05.487504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:05.488742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:05.489091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:05.490674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:05.493833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:05.519454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 838/1683] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:05 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:05.528590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:05.552328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ebcb7e-e416-48fd-aa63-712bd0a61b41 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:05.568926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:05.569765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:05.570042 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:05.570478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] PATH_INFO: `/v3/projects/e38dd123414b40c787869d00de4fb87b/users/2d1e93cb9f4742ae8cfb40370de44b22/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:05.631493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e38dd123414b40c787869d00de4fb87b, user_id=2d1e93cb9f4742ae8cfb40370de44b22, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:05.637393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:05.709353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:05.709789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 846/1684] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:05 2022] PUT /identity/v3/projects/e38dd123414b40c787869d00de4fb87b/users/2d1e93cb9f4742ae8cfb40370de44b22/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:05.755566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f245fddb-34e6-43b0-a8bc-c1f8aa792556 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:05.800825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:05.914049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f504e5c2-6257-4b4b-b5d4-b243586e8192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:05.915333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f504e5c2-6257-4b4b-b5d4-b243586e8192 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:05.915574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f504e5c2-6257-4b4b-b5d4-b243586e8192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:05.915851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f504e5c2-6257-4b4b-b5d4-b243586e8192 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:05.921312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f504e5c2-6257-4b4b-b5d4-b243586e8192 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:05.922974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f504e5c2-6257-4b4b-b5d4-b243586e8192 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:05.960993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:05.961797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 839/1685] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:05 2022] GET /identity/v3/roles => generated 1420 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:05.977352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfb3a46-f937-485e-8793-1584f48ff436 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:05.993359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:05.994235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:05.994550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:05.995055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] PATH_INFO: `/v3/projects/e38dd123414b40c787869d00de4fb87b/users/2d1e93cb9f4742ae8cfb40370de44b22/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:05.998781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e38dd123414b40c787869d00de4fb87b, user_id=2d1e93cb9f4742ae8cfb40370de44b22, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:06.006434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:06.056187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 847/1686] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:05 2022] PUT /identity/v3/projects/e38dd123414b40c787869d00de4fb87b/users/2d1e93cb9f4742ae8cfb40370de44b22/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:06.066537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300fd933-39a3-4161-853e-6a08ebae0b59 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:06.066795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300fd933-39a3-4161-853e-6a08ebae0b59 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:06.067078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300fd933-39a3-4161-853e-6a08ebae0b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:06.099596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:06.101812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:06.173739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-300fd933-39a3-4161-853e-6a08ebae0b59 None None] MFA Rules not processed for user `2d1e93cb9f4742ae8cfb40370de44b22`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:06.174975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-300fd933-39a3-4161-853e-6a08ebae0b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:06.376290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ec1f2b-d078-4429-965f-746035a1a0e4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:06.384264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:06.385075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:06.385358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:06.385634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] PATH_INFO: `/v3/users/2bd4786e1b444c318436ead072ca2541` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:06.386968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bd4786e1b444c318436ead072ca2541)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:06.389136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:06.423075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 840/1687] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:06 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 357 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:06.424156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a78c38-ce49-4466-a0fe-a6ce234a61de None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:06.424496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a78c38-ce49-4466-a0fe-a6ce234a61de None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:06.424713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a78c38-ce49-4466-a0fe-a6ce234a61de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:06.471767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-24a78c38-ce49-4466-a0fe-a6ce234a61de None None] MFA Rules not processed for user `dd95ef263d15460f9cca5a7557cbee2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:06.473191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-24a78c38-ce49-4466-a0fe-a6ce234a61de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:06.510900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bd4786e1b444c318436ead072ca2541'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:06.511920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bd4786e1b444c318436ead072ca2541'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:06.512927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bd4786e1b444c318436ead072ca2541'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:06.527198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bd4786e1b444c318436ead072ca2541'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:06.559682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=86,keystone:INSERT=5,keystone:DELETE=9 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:06.562672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bd4786e1b444c318436ead072ca2541'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:06.637344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 841/1688] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:06 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:06.639171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c8e66-5f81-4a9e-98a9-7d87d9d0ea69 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:06.640486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c8e66-5f81-4a9e-98a9-7d87d9d0ea69 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:06.640969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83c8e66-5f81-4a9e-98a9-7d87d9d0ea69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:06.651564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 848/1689] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:06 2022] DELETE /identity/v3/users/2bd4786e1b444c318436ead072ca2541 => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:06.654553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:06.658717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:06.808135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a401eece-571d-4e42-a5e2-1ae79b7b3c35 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:06.824331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83244a-d0c2-488b-9061-a31e19291ddf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:06.825269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83244a-d0c2-488b-9061-a31e19291ddf service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:06.825731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83244a-d0c2-488b-9061-a31e19291ddf service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:06.826248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83244a-d0c2-488b-9061-a31e19291ddf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:06.828380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ff83244a-d0c2-488b-9061-a31e19291ddf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:06.839728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e83c8e66-5f81-4a9e-98a9-7d87d9d0ea69 None None] MFA Rules not processed for user `7eb6ab0644bf4fa59d7bba232d6ccb3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:06.845293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e83c8e66-5f81-4a9e-98a9-7d87d9d0ea69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:06.935472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:06.988593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff83244a-d0c2-488b-9061-a31e19291ddf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:06.993523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff83244a-d0c2-488b-9061-a31e19291ddf service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:07.021927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 849/1690] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:06 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:07.024553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf9ad56-de7a-45ad-b7d8-84e25e1712fc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:07.026411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf9ad56-de7a-45ad-b7d8-84e25e1712fc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:07.026908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf9ad56-de7a-45ad-b7d8-84e25e1712fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:07.028662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 842/1691] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 390 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:07.030307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e3d7d-e7f8-41fd-88d4-9a50a8bc43c7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:07.031591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e3d7d-e7f8-41fd-88d4-9a50a8bc43c7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:07.032062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9e3d7d-e7f8-41fd-88d4-9a50a8bc43c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:07.088910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-be9e3d7d-e7f8-41fd-88d4-9a50a8bc43c7 None None] MFA Rules not processed for user `6fc9485505c443acb946ec66e0018f7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:07.090178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-be9e3d7d-e7f8-41fd-88d4-9a50a8bc43c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:07.155801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-baf9ad56-de7a-45ad-b7d8-84e25e1712fc None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:07.157326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-baf9ad56-de7a-45ad-b7d8-84e25e1712fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:07.229099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 843/1692] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:07.235054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c95f4c-2e9a-410c-a4dd-3cd1a5ef9706 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:07.235054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c95f4c-2e9a-410c-a4dd-3cd1a5ef9706 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:07.235054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c95f4c-2e9a-410c-a4dd-3cd1a5ef9706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:07.275659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e9c95f4c-2e9a-410c-a4dd-3cd1a5ef9706 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:07.276963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e9c95f4c-2e9a-410c-a4dd-3cd1a5ef9706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:07.300348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 850/1693] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:54:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:07.346911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:07.355398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:07.358045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:07.382969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 844/1694] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:07.387376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c95f4c-2e9a-410c-a4dd-3cd1a5ef9706 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:07.399691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c95f4c-2e9a-410c-a4dd-3cd1a5ef9706 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:07.405075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:07.405075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:07.405075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:07.405367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:07.407212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:07.417469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] MFA Rules not processed for user `7eb6ab0644bf4fa59d7bba232d6ccb3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:07.419321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:07.447783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 851/1695] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:07 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:07.449262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:07.452621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:07.470429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:07.530985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:07.536328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:07.574485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 845/1696] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:07 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:07.645991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dde625d-e49a-4998-9300-0ad3e1453bf1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:07.656561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:07.659858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:07.668567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd70a9a-5bb2-4bec-9aa8-72841dea8d37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:07.669301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd70a9a-5bb2-4bec-9aa8-72841dea8d37 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:07.675693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd70a9a-5bb2-4bec-9aa8-72841dea8d37 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:07.676113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd70a9a-5bb2-4bec-9aa8-72841dea8d37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:07.704295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:07.724006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd70a9a-5bb2-4bec-9aa8-72841dea8d37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:07.730788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd70a9a-5bb2-4bec-9aa8-72841dea8d37 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:07.790195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 852/1697] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:07.925918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a934276f-97e5-42e0-af19-6c86b7ba4a59 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:07.937518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9212349d-24d4-42f6-8f47-57c6578f6060 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:07.939792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9212349d-24d4-42f6-8f47-57c6578f6060 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:07.940141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9212349d-24d4-42f6-8f47-57c6578f6060 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:07.940502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9212349d-24d4-42f6-8f47-57c6578f6060 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:07.956768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9212349d-24d4-42f6-8f47-57c6578f6060 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:07.961989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9212349d-24d4-42f6-8f47-57c6578f6060 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:08.002312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 846/1698] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:54:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:54:09.082153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d2f9b3-6023-4282-b1ab-a5cfd2df84bd None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:09.082727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d2f9b3-6023-4282-b1ab-a5cfd2df84bd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:09.083317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d2f9b3-6023-4282-b1ab-a5cfd2df84bd None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:09.085622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 853/1699] 10.222.0.42 () {54 vars in 912 bytes} [Fri Sep 16 01:54:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:54:09.103617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e282c3b6-8d9b-4112-b761-0057ac6f6924 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:09.104162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e282c3b6-8d9b-4112-b761-0057ac6f6924 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:09.104514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e282c3b6-8d9b-4112-b761-0057ac6f6924 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:09.181140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e282c3b6-8d9b-4112-b761-0057ac6f6924 None None] MFA Rules not processed for user `6a343a260fa2415e84caf959933a6c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:09.184080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e282c3b6-8d9b-4112-b761-0057ac6f6924 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:09.251476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 847/1700] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:54:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:09.323737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d2f9b3-6023-4282-b1ab-a5cfd2df84bd None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:09.354485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d2f9b3-6023-4282-b1ab-a5cfd2df84bd None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:09.380403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:09.381448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:09.382129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:09.382554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:09.413177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:09.416246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:09.452743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 854/1701] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:10.131708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:10.133941 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:10.134560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:10.157127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:10.158885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:10.183929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] MFA Rules not processed for user `6a343a260fa2415e84caf959933a6c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:10.185262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:10.196390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b18c85d-7482-4790-bc15-57c7b84d8acb service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:10.203541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 848/1702] 10.222.0.50 () {58 vars in 1015 bytes} [Fri Sep 16 01:54:10 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:10.215848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:10.217614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:10.219691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:10.220887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:10.221410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:10.222403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] PATH_INFO: `/v3/users/167d5891f699468b915a620c1db66339` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:10.226028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] RBAC: Authorizing `identity:delete_user(user_id=167d5891f699468b915a620c1db66339)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:10.236087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:10.250138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1646ea2-a378-4abf-a384-e8ff95c2ff59 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:10.259348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:10.260249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:10.260736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:10.261203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:10.275624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:10.277503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:10.300260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 849/1703] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:10 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:10.304278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:10.308678 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:10.394762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '167d5891f699468b915a620c1db66339'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:10.396426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '167d5891f699468b915a620c1db66339'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:10.408016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '167d5891f699468b915a620c1db66339'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:10.425807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '167d5891f699468b915a620c1db66339'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:10.483528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-52a883b5-c0ae-44ff-82ed-4e100eb71953 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '167d5891f699468b915a620c1db66339'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:10.507159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 855/1704] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:10 2022] DELETE /identity/v3/users/167d5891f699468b915a620c1db66339 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:10.513628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:10.516565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:10.517816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:10.571130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:10.571620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9bfd7a-1c24-440e-8e97-0ce7158e113c service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:10.573459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:10.696243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 856/1705] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:10.761162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:10.763125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:10.784486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:10.836853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:10.873817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:10.874876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:10.875357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:10.875827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:10.884648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:11.073037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:11.090099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:11.091456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ae0102-f915-4868-a43f-60d91ace971b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:11.129346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2076429-12a5-47dd-8502-c57252bf5aa1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:11.130497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2076429-12a5-47dd-8502-c57252bf5aa1 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:11.132451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2076429-12a5-47dd-8502-c57252bf5aa1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:11.132855 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2076429-12a5-47dd-8502-c57252bf5aa1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:11.143082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 850/1706] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:10 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 843 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:11.148919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:11.152760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:11.174819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2076429-12a5-47dd-8502-c57252bf5aa1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:11.182055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2076429-12a5-47dd-8502-c57252bf5aa1 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:11.249674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bb4b71-ddf5-4536-97db-4e440e650413 service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:11.250010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 857/1707] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:11.307145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:11.308238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:11.321176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:11.321527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] PATH_INFO: `/v3/projects/7b338918fdb447db92679939b9f534c4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:11.326225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b338918fdb447db92679939b9f534c4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:11.332979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:11.469053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b338918fdb447db92679939b9f534c4', 'request_id': 'req-8c9498b0-529f-4804-8aec-a5e577a897cf'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:11.487760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b338918fdb447db92679939b9f534c4', 'request_id': 'req-8c9498b0-529f-4804-8aec-a5e577a897cf'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:11.529123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b338918fdb447db92679939b9f534c4', 'request_id': 'req-8c9498b0-529f-4804-8aec-a5e577a897cf'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:11.575783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] The token cache is being invalidate because project 7b338918fdb447db92679939b9f534c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:11.576366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:11.582542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8c9498b0-529f-4804-8aec-a5e577a897cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:11.592593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 851/1708] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:11 2022] DELETE /identity/v3/projects/7b338918fdb447db92679939b9f534c4 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:11.593282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effd92c-2c47-44e7-80bd-4bf834308f2e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:11.593743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effd92c-2c47-44e7-80bd-4bf834308f2e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:11.594174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1effd92c-2c47-44e7-80bd-4bf834308f2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:11.662642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1effd92c-2c47-44e7-80bd-4bf834308f2e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:11.677013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1effd92c-2c47-44e7-80bd-4bf834308f2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:12.071524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecffc49-c646-4102-85e6-6f366a427eb8 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:12.071756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecffc49-c646-4102-85e6-6f366a427eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:12.072064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecffc49-c646-4102-85e6-6f366a427eb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:12.079140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 858/1709] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 487 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:12.169053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5ecffc49-c646-4102-85e6-6f366a427eb8 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:12.170640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ecffc49-c646-4102-85e6-6f366a427eb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:12.201703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effd92c-2c47-44e7-80bd-4bf834308f2e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:12.222365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 852/1710] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:12.229061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1effd92c-2c47-44e7-80bd-4bf834308f2e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:12.236038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:12.236743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:12.237039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:12.237379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:12.238170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:12.238472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:12.240484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:12.292236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecffc49-c646-4102-85e6-6f366a427eb8 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:12.307856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 859/1711] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:12.313207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecffc49-c646-4102-85e6-6f366a427eb8 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:12.327219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:12.327852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:12.328184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:12.328450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:12.332974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:12.338753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:12.435088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:12.446903 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 853/1712] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:12 2022] POST /identity/v3/projects => generated 367 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:12.455422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793948ca-f321-4f8e-8674-01a7db77d15d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:12.677477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:12.678307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:12.678803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:12.679483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:12.680329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:12.686004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:12.721256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:12.724035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:12.757090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 860/1713] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:12 2022] POST /identity/v3/users => generated 284 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:12.769072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:12.804211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a2826-88a1-42a4-bf52-e8b1e964175a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:12.812482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cf30a4-8340-476f-8c7c-abaa4f456ea0 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:12.822529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:12.823287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:12.823673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:12.824076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:12.824837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:12.826560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:12.845053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:12.848109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:12.849378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:12.850077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] PATH_INFO: `/v3/projects/7b8c9cc2c53d43a895e570024b80d98b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:12.858964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b8c9cc2c53d43a895e570024b80d98b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:12.863391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:12.863529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 861/1714] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:12 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:12.873760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:12.916334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef1badf6-c962-4432-9480-f2fd2942ee1a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:12.936889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:12.937655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:12.938162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:12.938263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:12.939457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:12.942528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:12.980172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 862/1715] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:12 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:12.982577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:13.010366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:13.037185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d9a161-fb6f-455a-8c19-e7b2dddec3fd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:13.048520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:13.052721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b8c9cc2c53d43a895e570024b80d98b', 'request_id': 'req-03d26596-7731-4e03-8874-418fb9c5e13b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.080167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b8c9cc2c53d43a895e570024b80d98b', 'request_id': 'req-03d26596-7731-4e03-8874-418fb9c5e13b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.089291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b8c9cc2c53d43a895e570024b80d98b', 'request_id': 'req-03d26596-7731-4e03-8874-418fb9c5e13b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.121065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] The token cache is being invalidate because project 7b8c9cc2c53d43a895e570024b80d98b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:13.121357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.135450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.139555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 854/1716] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:12 2022] DELETE /identity/v3/projects/7b8c9cc2c53d43a895e570024b80d98b => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:13.148763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:13.148763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:13.255413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af3022c-1bb7-4329-baec-38046999332b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:13.256260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af3022c-1bb7-4329-baec-38046999332b admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:13.257087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af3022c-1bb7-4329-baec-38046999332b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:13.257295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af3022c-1bb7-4329-baec-38046999332b admin admin] PATH_INFO: `/v3/projects/88f961f25de64533bf83beede20089cf/users/50695b030d2d401680f7862f823a5ad6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:13.305836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af3022c-1bb7-4329-baec-38046999332b admin admin] RBAC: Authorizing `identity:create_grant(project_id=88f961f25de64533bf83beede20089cf, user_id=50695b030d2d401680f7862f823a5ad6, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:13.308673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af3022c-1bb7-4329-baec-38046999332b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:13.310499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d26596-7731-4e03-8874-418fb9c5e13b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:13.318627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:13.320224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:13.320833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:13.321261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] PATH_INFO: `/v3/users/3ab0fcd27cbb473b87793afe15784e48` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:13.324518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ab0fcd27cbb473b87793afe15784e48)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:13.326830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:13.371217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 863/1717] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:12 2022] PUT /identity/v3/projects/88f961f25de64533bf83beede20089cf/users/50695b030d2d401680f7862f823a5ad6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:13.372830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d564aec-b50b-4845-b8fa-b384aa60f092 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:13.373123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d564aec-b50b-4845-b8fa-b384aa60f092 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:13.373397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d564aec-b50b-4845-b8fa-b384aa60f092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:13.407900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ab0fcd27cbb473b87793afe15784e48'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.408813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ab0fcd27cbb473b87793afe15784e48'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.427556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ab0fcd27cbb473b87793afe15784e48'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.429617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ab0fcd27cbb473b87793afe15784e48'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.453385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5d564aec-b50b-4845-b8fa-b384aa60f092 None None] MFA Rules not processed for user `50695b030d2d401680f7862f823a5ad6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:13.454603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5d564aec-b50b-4845-b8fa-b384aa60f092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:13.462855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ab0fcd27cbb473b87793afe15784e48'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:13.499682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 855/1718] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:13 2022] DELETE /identity/v3/users/3ab0fcd27cbb473b87793afe15784e48 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:13.570157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 864/1719] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:13 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:13.584033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:13.586343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:13.694598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d559a7-2d70-4cfd-a6ef-c132172170a9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:13.703414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61467ef-4655-422e-81b2-22e067468352 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:13.704337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61467ef-4655-422e-81b2-22e067468352 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:13.705508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61467ef-4655-422e-81b2-22e067468352 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:13.706069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61467ef-4655-422e-81b2-22e067468352 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:13.722660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:13.801962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61467ef-4655-422e-81b2-22e067468352 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:13.804323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61467ef-4655-422e-81b2-22e067468352 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:13.833357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 856/1720] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:13 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:14.009696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.009696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.009966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.032824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3739de9b-31fa-4acb-b68b-1991897d33fa None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.034347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3739de9b-31fa-4acb-b68b-1991897d33fa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.034696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3739de9b-31fa-4acb-b68b-1991897d33fa None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.036837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 857/1721] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:54:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:54:14.051458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.051742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.052041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.080312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] MFA Rules not processed for user `7eb6ab0644bf4fa59d7bba232d6ccb3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:14.082331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:14.114089 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:14.117813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:14.254290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 865/1722] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:14 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:14.255646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:14.258171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:14.336790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:14.392850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 858/1723] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:54:14 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 342 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:14.403993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ce12b-3f55-4f80-bbe9-e9821c9301eb None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:14.409165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:14.410041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:14.410823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.411115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:14.411367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.411732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] PATH_INFO: `/v3/projects/e1de48fb8c9542e399b86c1d76151b00` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.421486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1de48fb8c9542e399b86c1d76151b00)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:14.423894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:14.449581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccdcd95-2a37-41cc-ad0e-a95e3e150267 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:14.467384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc751805-044b-41f5-81f5-07fd9ecb5974 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:14.468050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc751805-044b-41f5-81f5-07fd9ecb5974 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.468467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc751805-044b-41f5-81f5-07fd9ecb5974 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.468747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc751805-044b-41f5-81f5-07fd9ecb5974 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.496855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc751805-044b-41f5-81f5-07fd9ecb5974 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:14.498483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc751805-044b-41f5-81f5-07fd9ecb5974 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:14.517358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 859/1724] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:14 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:14.560522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1de48fb8c9542e399b86c1d76151b00', 'request_id': 'req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:14.573172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1de48fb8c9542e399b86c1d76151b00', 'request_id': 'req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:14.581748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1de48fb8c9542e399b86c1d76151b00', 'request_id': 'req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:14.604010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] The token cache is being invalidate because project e1de48fb8c9542e399b86c1d76151b00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:14.607605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:14.609032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-92a28fb6-97c0-440e-acf9-5b9d5bde2dcc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:14.613081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 866/1725] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:14 2022] DELETE /identity/v3/projects/e1de48fb8c9542e399b86c1d76151b00 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:14.619086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238f4101-2490-4eed-a5d9-d1afeac543df None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.619833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238f4101-2490-4eed-a5d9-d1afeac543df None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.620337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238f4101-2490-4eed-a5d9-d1afeac543df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.652820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-238f4101-2490-4eed-a5d9-d1afeac543df None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:14.654038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-238f4101-2490-4eed-a5d9-d1afeac543df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:14.730098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 860/1726] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:14.739900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f40237a-7f00-4bbc-909b-6d3dae647580 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.740295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f40237a-7f00-4bbc-909b-6d3dae647580 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.740603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f40237a-7f00-4bbc-909b-6d3dae647580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.767615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5f40237a-7f00-4bbc-909b-6d3dae647580 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:14.768898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5f40237a-7f00-4bbc-909b-6d3dae647580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:14.778293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 867/1727] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:14.782925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238f4101-2490-4eed-a5d9-d1afeac543df None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:14.793065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238f4101-2490-4eed-a5d9-d1afeac543df None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:14.797762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:14.798361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.798635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.798900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.799541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:14.799834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:14.801453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:14.815545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 861/1728] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:14.820368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f40237a-7f00-4bbc-909b-6d3dae647580 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:14.830005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f40237a-7f00-4bbc-909b-6d3dae647580 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:14.834747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:14.835388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.835663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.835958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.836592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:14.838518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:14.868601 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 868/1729] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:14 2022] POST /identity/v3/projects => generated 363 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:14.872951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:14.883402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1753f206-e6c9-4f09-a33f-b1b5a827a098 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:14.896450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:14.949831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:14.950555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:14.950899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:14.951236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:14.951962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:14.953955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:14.981616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 862/1730] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:14 2022] POST /identity/v3/users => generated 282 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:14.982532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:14.995138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ae1842-5b63-4969-8fbe-aa08b446e0fe admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:14.999564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34933733-9608-48fb-a637-79551731735b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.000280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34933733-9608-48fb-a637-79551731735b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.000628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34933733-9608-48fb-a637-79551731735b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.002102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34933733-9608-48fb-a637-79551731735b admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.002102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34933733-9608-48fb-a637-79551731735b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.003299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34933733-9608-48fb-a637-79551731735b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.015527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 869/1731] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:15.019792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.039394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfc3360-ad25-4445-bcda-f020c684a8df admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.046456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d36e1-ef58-4c3e-89bc-53a2b20aa8a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.047156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d36e1-ef58-4c3e-89bc-53a2b20aa8a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.047492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d36e1-ef58-4c3e-89bc-53a2b20aa8a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.048359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d36e1-ef58-4c3e-89bc-53a2b20aa8a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.049929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d36e1-ef58-4c3e-89bc-53a2b20aa8a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.052766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d36e1-ef58-4c3e-89bc-53a2b20aa8a9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.090674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 863/1732] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:15.092023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34933733-9608-48fb-a637-79551731735b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.106637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34933733-9608-48fb-a637-79551731735b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.116982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.117831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.118217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.118592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] PATH_INFO: `/v3/projects/7041cdc228e349768b1307b09fce4c90/users/d770244bc4d845c7a15ae44a136620ad/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.149814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7041cdc228e349768b1307b09fce4c90, user_id=d770244bc4d845c7a15ae44a136620ad, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.153069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.178976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 870/1733] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:15 2022] PUT /identity/v3/projects/7041cdc228e349768b1307b09fce4c90/users/d770244bc4d845c7a15ae44a136620ad/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:15.189150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941220f6-7b80-4199-93ef-bd7538f74636 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.189432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941220f6-7b80-4199-93ef-bd7538f74636 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.189704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941220f6-7b80-4199-93ef-bd7538f74636 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.234658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-941220f6-7b80-4199-93ef-bd7538f74636 None None] MFA Rules not processed for user `d770244bc4d845c7a15ae44a136620ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:15.235955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-941220f6-7b80-4199-93ef-bd7538f74636 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:15.308421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 864/1734] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:15 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:15.312656 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.382265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:15.402596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febe7771-b83a-4aad-8f33-8bd33295a283 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.409126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.410597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.411235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.411509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.412212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.415315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.448751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 871/1735] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:15 2022] POST /identity/v3/projects => generated 365 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:54:15.453609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941220f6-7b80-4199-93ef-bd7538f74636 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.477638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941220f6-7b80-4199-93ef-bd7538f74636 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.545050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.545669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.545988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.546257 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.546884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.548881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.578854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 865/1736] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:15 2022] POST /identity/v3/users => generated 289 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:15.588461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.602328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb2d507-cdcd-4cd4-88c3-6cf20bf9edfb admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.609513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.610349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.610776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.611212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.612177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.613895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.628297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 872/1737] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:15.633750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.647685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d6fe7d-b5ea-44f9-a8e2-aec76dbedfbe admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.652770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.653515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.653868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.654218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] PATH_INFO: `/v3/projects/68529af86021494f85fa7b2fab72f508/users/080a1c84f9024a7dab2acedbccd34d07/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.668455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:15.674014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=68529af86021494f85fa7b2fab72f508, user_id=080a1c84f9024a7dab2acedbccd34d07, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.677424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.692695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 866/1738] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:15 2022] PUT /identity/v3/projects/68529af86021494f85fa7b2fab72f508/users/080a1c84f9024a7dab2acedbccd34d07/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:15.697469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.707651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e074dfd2-bdd1-4c6b-8d8f-ede1342cc0c4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.779322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31aa903-b0d4-41af-ac1d-0fc3a8e63d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.779905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31aa903-b0d4-41af-ac1d-0fc3a8e63d7e admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.780257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31aa903-b0d4-41af-ac1d-0fc3a8e63d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.780600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31aa903-b0d4-41af-ac1d-0fc3a8e63d7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.781293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31aa903-b0d4-41af-ac1d-0fc3a8e63d7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.782884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31aa903-b0d4-41af-ac1d-0fc3a8e63d7e admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.793214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 873/1739] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:15.797529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.809893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a5d66-bb35-4448-b2f2-cfacbb87f7d2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:15.814656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:15.815315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.815594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.815881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] PATH_INFO: `/v3/projects/68529af86021494f85fa7b2fab72f508/users/080a1c84f9024a7dab2acedbccd34d07/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.817907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] RBAC: Authorizing `identity:create_grant(project_id=68529af86021494f85fa7b2fab72f508, user_id=080a1c84f9024a7dab2acedbccd34d07, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:15.821222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:15.832186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 867/1740] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:15 2022] PUT /identity/v3/projects/68529af86021494f85fa7b2fab72f508/users/080a1c84f9024a7dab2acedbccd34d07/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:15.843375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04334c6-c0ed-4935-a68b-b8d6e3382eab None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:15.843664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04334c6-c0ed-4935-a68b-b8d6e3382eab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:15.843977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04334c6-c0ed-4935-a68b-b8d6e3382eab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:15.874831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b04334c6-c0ed-4935-a68b-b8d6e3382eab None None] MFA Rules not processed for user `080a1c84f9024a7dab2acedbccd34d07`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:15.876159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b04334c6-c0ed-4935-a68b-b8d6e3382eab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:15.955528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 874/1741] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:15 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:15.967459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:15.969702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:16.034333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea624379-caa4-49d4-8c3a-263cde6820be admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:16.038198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:16.038997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:16.039271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:16.039532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:16.049261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:16.050709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:16.061716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 868/1742] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:15 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:16.139970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04334c6-c0ed-4935-a68b-b8d6e3382eab None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:16.150858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04334c6-c0ed-4935-a68b-b8d6e3382eab None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:16.157003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3002eed-0e7c-4b79-a437-828535e87a71 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:16.157707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3002eed-0e7c-4b79-a437-828535e87a71 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:16.157889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3002eed-0e7c-4b79-a437-828535e87a71 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:16.158158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3002eed-0e7c-4b79-a437-828535e87a71 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:16.226416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:16.250341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3002eed-0e7c-4b79-a437-828535e87a71 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:16.252358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3002eed-0e7c-4b79-a437-828535e87a71 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:16.265195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 875/1743] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:16 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:20.765215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:20.909145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb3823d-4b39-4f7a-8622-8ae0b1cb7453 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:20.916381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:20.917041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:20.917330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:20.917723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] PATH_INFO: `/v3/users/d770244bc4d845c7a15ae44a136620ad` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:20.919007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d770244bc4d845c7a15ae44a136620ad)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:20.921756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:21.025260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd770244bc4d845c7a15ae44a136620ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:21.026010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd770244bc4d845c7a15ae44a136620ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:21.045009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd770244bc4d845c7a15ae44a136620ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:21.047301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd770244bc4d845c7a15ae44a136620ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:21.094990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd770244bc4d845c7a15ae44a136620ad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:21.110476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:21.123281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 869/1744] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:20 2022] DELETE /identity/v3/users/d770244bc4d845c7a15ae44a136620ad => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:21.128963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:21.129595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:21.129595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:21.181301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:21.184619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:21.300524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 876/1745] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:21.310014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:21.312318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:21.512974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5820e475-18fe-44c0-9367-2fb6c306d7d8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:21.521005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:21.521995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:21.522473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:21.522817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:21.540265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:21.541955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:21.566008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 870/1746] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:21.970980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:21.973317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:22.043210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44ed3a3-5345-4329-b40c-49598083a7cb None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:22.056689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:22.057451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:22.057759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:22.058708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] PATH_INFO: `/v3/projects/7041cdc228e349768b1307b09fce4c90` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:22.064313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7041cdc228e349768b1307b09fce4c90)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:22.066960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:22.139357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7041cdc228e349768b1307b09fce4c90', 'request_id': 'req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.145837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7041cdc228e349768b1307b09fce4c90', 'request_id': 'req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.151590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7041cdc228e349768b1307b09fce4c90', 'request_id': 'req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.166140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] The token cache is being invalidate because project 7041cdc228e349768b1307b09fce4c90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:22.166477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.172479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.175095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 877/1747] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:21 2022] DELETE /identity/v3/projects/7041cdc228e349768b1307b09fce4c90 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:22.180160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:22.201942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62c5be0-0578-4e30-bd39-9227e41e740d service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:22.267715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:22.268383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:22.268662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:22.268933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] PATH_INFO: `/v3/users/080a1c84f9024a7dab2acedbccd34d07` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:22.270145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] RBAC: Authorizing `identity:delete_user(user_id=080a1c84f9024a7dab2acedbccd34d07)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:22.272472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:22.344750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '080a1c84f9024a7dab2acedbccd34d07'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.345474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '080a1c84f9024a7dab2acedbccd34d07'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.359472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '080a1c84f9024a7dab2acedbccd34d07'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.360232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '080a1c84f9024a7dab2acedbccd34d07'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.375427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-420ceb9c-dc93-4da1-8802-4d3b765d763d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '080a1c84f9024a7dab2acedbccd34d07'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:22.397239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 871/1748] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:22 2022] DELETE /identity/v3/users/080a1c84f9024a7dab2acedbccd34d07 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:22.804015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:22.812179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:22.837131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3e4c4-4bcb-4b2b-a09b-7b8d815f29a3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:22.953049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:22.953944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:22.954384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:22.954829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] PATH_INFO: `/v3/projects/68529af86021494f85fa7b2fab72f508` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:22.956547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=68529af86021494f85fa7b2fab72f508)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:22.958784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:22.973572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:23.038990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '68529af86021494f85fa7b2fab72f508', 'request_id': 'req-1b52dce0-e182-4954-b298-fcf91a1801bf'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:23.050807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '68529af86021494f85fa7b2fab72f508', 'request_id': 'req-1b52dce0-e182-4954-b298-fcf91a1801bf'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:23.059968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '68529af86021494f85fa7b2fab72f508', 'request_id': 'req-1b52dce0-e182-4954-b298-fcf91a1801bf'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:23.081914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] The token cache is being invalidate because project 68529af86021494f85fa7b2fab72f508 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:23.082215 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:23.083332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1b52dce0-e182-4954-b298-fcf91a1801bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:23.092054 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 878/1749] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:22 2022] DELETE /identity/v3/projects/68529af86021494f85fa7b2fab72f508 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:23.094729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de28a73-f456-406b-ae00-70dde6fbe5ad None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.095024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de28a73-f456-406b-ae00-70dde6fbe5ad None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.095295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de28a73-f456-406b-ae00-70dde6fbe5ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.117685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0de28a73-f456-406b-ae00-70dde6fbe5ad None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:23.118898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0de28a73-f456-406b-ae00-70dde6fbe5ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:23.143340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:23.210353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 872/1750] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:23.218082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d0f28e-fd3e-43a6-bc4b-4c9d093adc7f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.218942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d0f28e-fd3e-43a6-bc4b-4c9d093adc7f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.219411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d0f28e-fd3e-43a6-bc4b-4c9d093adc7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.245756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b8d0f28e-fd3e-43a6-bc4b-4c9d093adc7f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:23.247444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b8d0f28e-fd3e-43a6-bc4b-4c9d093adc7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:23.274459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 879/1751] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:23.279601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de28a73-f456-406b-ae00-70dde6fbe5ad None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.294186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de28a73-f456-406b-ae00-70dde6fbe5ad None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.299259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:23.299894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.300220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.300490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.301139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:23.301584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:23.304408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:23.325042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 873/1752] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:23.329528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d0f28e-fd3e-43a6-bc4b-4c9d093adc7f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.344227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d0f28e-fd3e-43a6-bc4b-4c9d093adc7f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.365359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:23.366059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.366414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.366544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.367183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:23.369148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:23.421097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 880/1753] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:23 2022] POST /identity/v3/projects => generated 407 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:23.422444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.436261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ff682f-459a-4421-b28d-137ab7308604 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.508246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:23.509133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.509566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.509990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.510778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:23.512899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:23.541957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 874/1754] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:23 2022] POST /identity/v3/users => generated 304 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:23.546628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.566698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c32224-b4b8-4142-8231-cab776a0238d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.574039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:23.575065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.575358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.576068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.577585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:23.581291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:23.594860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 881/1755] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:23.599246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.611409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3927752-3483-4003-bbf6-c64351c0225c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.623505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcca45e7-a5fd-4497-902c-55b019394d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:23.624199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca45e7-a5fd-4497-902c-55b019394d19 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.624490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca45e7-a5fd-4497-902c-55b019394d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.624683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca45e7-a5fd-4497-902c-55b019394d19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.625415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcca45e7-a5fd-4497-902c-55b019394d19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:23.627024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcca45e7-a5fd-4497-902c-55b019394d19 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:23.639293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 875/1756] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:23 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:23.643472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.653790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f613b9-9338-41f1-80f5-c429ceb86db2 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.658456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:23.659105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.659390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.659664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] PATH_INFO: `/v3/projects/f43b9cc8d197448faab988d13724667f/users/552197ac405a44ca8ae03201cd2a1245/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.676110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] RBAC: Authorizing `identity:create_grant(project_id=f43b9cc8d197448faab988d13724667f, user_id=552197ac405a44ca8ae03201cd2a1245, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:23.689377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:23.700774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 882/1757] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:23 2022] PUT /identity/v3/projects/f43b9cc8d197448faab988d13724667f/users/552197ac405a44ca8ae03201cd2a1245/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:23.712340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3436ac51-f519-4694-a386-a95fc1c68f3b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.712633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3436ac51-f519-4694-a386-a95fc1c68f3b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.712905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3436ac51-f519-4694-a386-a95fc1c68f3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.742642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3436ac51-f519-4694-a386-a95fc1c68f3b None None] MFA Rules not processed for user `552197ac405a44ca8ae03201cd2a1245`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:23.744381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3436ac51-f519-4694-a386-a95fc1c68f3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:23.819687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 876/1758] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:23 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:23.824821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.891651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89445331-3eea-4bea-80d0-3a98b5548cac admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.896383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:23.897090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:23.897368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:23.897641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:23.898261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:23.900228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:23.934889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 883/1759] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:23 2022] POST /identity/v3/projects => generated 409 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:23.939139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3436ac51-f519-4694-a386-a95fc1c68f3b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:23.950026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3436ac51-f519-4694-a386-a95fc1c68f3b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:23.962984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:24.015406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.016162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.016465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.016735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.017401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.019650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.042639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 877/1760] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:23 2022] POST /identity/v3/users => generated 311 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:54:24.046832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:24.058523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea125898-fdc9-4008-bb82-e7f6bb240299 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:24.063498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.064170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.064507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.064875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.065518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.067196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.079354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 884/1761] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:24 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:24.085597 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:24.095507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78311ff1-8fef-4ce2-a7bf-8e709956cd4d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:24.101802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.102470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.102851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.103064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] PATH_INFO: `/v3/projects/ffb50e3d06bb41daab42057d2dfc0f05/users/28eb794b7f264d2db783a898d0db9e2d/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.119244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffb50e3d06bb41daab42057d2dfc0f05, user_id=28eb794b7f264d2db783a898d0db9e2d, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.122078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.135827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 878/1762] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:24 2022] PUT /identity/v3/projects/ffb50e3d06bb41daab42057d2dfc0f05/users/28eb794b7f264d2db783a898d0db9e2d/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:24.141041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:24.152655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ec5e02-d63c-465d-b244-27374ec89ddf admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:24.157055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:24.223737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d0ccbc-e7f1-4160-90c5-aa8af599acde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.224565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d0ccbc-e7f1-4160-90c5-aa8af599acde admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.225573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d0ccbc-e7f1-4160-90c5-aa8af599acde admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.226110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d0ccbc-e7f1-4160-90c5-aa8af599acde admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.227052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d0ccbc-e7f1-4160-90c5-aa8af599acde admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.228960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d0ccbc-e7f1-4160-90c5-aa8af599acde admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.240006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 885/1763] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:24 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:24.244332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:24.255211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c148c1d9-9ded-4225-aad0-c134eb64a3bd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:24.261074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.262059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.263060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.263346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] PATH_INFO: `/v3/projects/ffb50e3d06bb41daab42057d2dfc0f05/users/28eb794b7f264d2db783a898d0db9e2d/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.266500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffb50e3d06bb41daab42057d2dfc0f05, user_id=28eb794b7f264d2db783a898d0db9e2d, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.269721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.282714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 879/1764] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:24 2022] PUT /identity/v3/projects/ffb50e3d06bb41daab42057d2dfc0f05/users/28eb794b7f264d2db783a898d0db9e2d/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:24.294316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.294657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.294940 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.324326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] MFA Rules not processed for user `28eb794b7f264d2db783a898d0db9e2d`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:24.325595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:24.407144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 886/1765] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:24 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:24.420596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:24.422528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:24.500208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893a50d4-4870-41d8-b54a-c7043f540ffb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:24.504233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.505205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.505603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.506490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.525757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.527185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.537774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 880/1766] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:24 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:24.698328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:24.699854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:24.753794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:24.754448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:24.782891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbd7d60-6724-4bd7-bf3a-f75301178b9d service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:24.812541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d294ec-a25e-4feb-ab46-b7b0e50b8ee5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.814164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d294ec-a25e-4feb-ab46-b7b0e50b8ee5 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.814643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d294ec-a25e-4feb-ab46-b7b0e50b8ee5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.815155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d294ec-a25e-4feb-ab46-b7b0e50b8ee5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.870219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7330248a-1220-456a-a2e2-1b5d2acfee18 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:24.876877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:24.877614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:24.877933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:24.878150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] PATH_INFO: `/v3/users/cc17bab3e54b42a7b367919371a07c8f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:24.879320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cc17bab3e54b42a7b367919371a07c8f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.882296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.925265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d294ec-a25e-4feb-ab46-b7b0e50b8ee5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:24.927171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d294ec-a25e-4feb-ab46-b7b0e50b8ee5 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:24.957988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 881/1767] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:24 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:24.966718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:25.012968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cc17bab3e54b42a7b367919371a07c8f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:25.016389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cc17bab3e54b42a7b367919371a07c8f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:25.017207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cc17bab3e54b42a7b367919371a07c8f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:25.043050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cc17bab3e54b42a7b367919371a07c8f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:25.062619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cc17bab3e54b42a7b367919371a07c8f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:25.083767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 887/1768] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:24 2022] DELETE /identity/v3/users/cc17bab3e54b42a7b367919371a07c8f => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:25.092966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:25.093295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:25.093573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:25.120749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:25.122721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:25.128951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:25.219512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 882/1769] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:25.230369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:25.232994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:25.339976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50327f0-a6cd-4f84-8f01-b5333d7526a0 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:25.346200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:25.346885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:25.347165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:25.347488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:25.362033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:25.364117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:25.376323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 888/1770] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:26.642972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:26.645720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:26.691558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1b1f40-38f4-416b-bd84-a8f3d21a271a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:26.699992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:26.700808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:26.701819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:26.702606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] PATH_INFO: `/v3/projects/45db59daaa1c42e689bb38c8e6ce6820` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:26.705191 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45db59daaa1c42e689bb38c8e6ce6820)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:26.707488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:26.866220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45db59daaa1c42e689bb38c8e6ce6820', 'request_id': 'req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:26.873348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45db59daaa1c42e689bb38c8e6ce6820', 'request_id': 'req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:26.880725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45db59daaa1c42e689bb38c8e6ce6820', 'request_id': 'req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:26.933426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] The token cache is being invalidate because project 45db59daaa1c42e689bb38c8e6ce6820 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:26.933784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:26.944461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:26.948600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 883/1771] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:26 2022] DELETE /identity/v3/projects/45db59daaa1c42e689bb38c8e6ce6820 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:26.952313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:26.968292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8faa0c95-0788-4682-9b1b-ee086273ba0d service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:27.093430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:27.094417 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:27.094845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:27.095260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] PATH_INFO: `/v3/users/9640ac96ecee44f88793bcc79c5c630b` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:27.096597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9640ac96ecee44f88793bcc79c5c630b)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:27.098742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:27.209529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9640ac96ecee44f88793bcc79c5c630b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.210682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9640ac96ecee44f88793bcc79c5c630b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.212527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9640ac96ecee44f88793bcc79c5c630b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.219523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:27.225931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9640ac96ecee44f88793bcc79c5c630b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.247538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9b3c65d2-9328-4c12-abee-fc05d396e81b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9640ac96ecee44f88793bcc79c5c630b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.252534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:27.254141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:27.276767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ccaeb-bacf-4377-a220-e609c3d81cc5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:27.282187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 889/1772] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:26 2022] DELETE /identity/v3/users/9640ac96ecee44f88793bcc79c5c630b => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:27.289327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:27.290688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:27.291148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:27.291601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] PATH_INFO: `/v3/users/50695b030d2d401680f7862f823a5ad6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:27.293188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=50695b030d2d401680f7862f823a5ad6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:27.295849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:27.375064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50695b030d2d401680f7862f823a5ad6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.375674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50695b030d2d401680f7862f823a5ad6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.385325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50695b030d2d401680f7862f823a5ad6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.386500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50695b030d2d401680f7862f823a5ad6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.403313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50695b030d2d401680f7862f823a5ad6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.420991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 884/1773] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:27 2022] DELETE /identity/v3/users/50695b030d2d401680f7862f823a5ad6 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:27.425511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:27.425792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:27.426073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:27.446212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:27.447381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:27.517985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 890/1774] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:27.545690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:27.548006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:27.560179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:27.632860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2f5733-31e4-4b86-9c0e-570a8138d3d8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:27.636524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440e4420-6ffd-4ded-aa39-a663add27529 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:27.637129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440e4420-6ffd-4ded-aa39-a663add27529 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:27.637408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440e4420-6ffd-4ded-aa39-a663add27529 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:27.637675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440e4420-6ffd-4ded-aa39-a663add27529 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:27.648075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440e4420-6ffd-4ded-aa39-a663add27529 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:27.649529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440e4420-6ffd-4ded-aa39-a663add27529 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:27.660209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 885/1775] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:27.775570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:27.777485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:27.816036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a0ed6-38d0-46a8-ab74-93d65c62e74e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:27.821362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:27.822380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:27.822749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:27.823090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] PATH_INFO: `/v3/projects/89bbdf6516124d80877954950aeeab29` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:27.824381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] RBAC: Authorizing `identity:delete_project(project_id=89bbdf6516124d80877954950aeeab29)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:27.826636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:27.917219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89bbdf6516124d80877954950aeeab29', 'request_id': 'req-cdf17fdc-5d98-47a3-87ff-070ebb836746'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.931761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89bbdf6516124d80877954950aeeab29', 'request_id': 'req-cdf17fdc-5d98-47a3-87ff-070ebb836746'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:27.958624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89bbdf6516124d80877954950aeeab29', 'request_id': 'req-cdf17fdc-5d98-47a3-87ff-070ebb836746'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.016976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] The token cache is being invalidate because project 89bbdf6516124d80877954950aeeab29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:28.017418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.018833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cdf17fdc-5d98-47a3-87ff-070ebb836746 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.023975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 891/1776] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:27 2022] DELETE /identity/v3/projects/89bbdf6516124d80877954950aeeab29 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:28.029003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7ea63e-dca8-45ec-9640-d7087da7b9b8 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.029629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7ea63e-dca8-45ec-9640-d7087da7b9b8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.030081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7ea63e-dca8-45ec-9640-d7087da7b9b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.063485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ee7ea63e-dca8-45ec-9640-d7087da7b9b8 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:28.065017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ee7ea63e-dca8-45ec-9640-d7087da7b9b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:28.158085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.158375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.158655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.163656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 886/1777] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:28.200422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:28.201767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:28.220764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 892/1778] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:28.225455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7ea63e-dca8-45ec-9640-d7087da7b9b8 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.237582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7ea63e-dca8-45ec-9640-d7087da7b9b8 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.245898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.246529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.246825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.247112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.247739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:28.252237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.255621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.257470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.259572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:28.275111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 887/1779] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:54:28.276097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.281413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46d2e09-9917-4c0c-96ec-803cc78a72ce None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.288199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52768ac8-d036-4005-bbee-a53336ab7137 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.290095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.290926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.291346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.291760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] PATH_INFO: `/v3/projects/88f961f25de64533bf83beede20089cf` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.293189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.293708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88f961f25de64533bf83beede20089cf)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.293838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.294044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.294915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.295477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.297604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.301907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.342867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 888/1780] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/projects => generated 391 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:28.343914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.367658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:28.381636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88f961f25de64533bf83beede20089cf', 'request_id': 'req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.397432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88f961f25de64533bf83beede20089cf', 'request_id': 'req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.412953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88f961f25de64533bf83beede20089cf', 'request_id': 'req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.435273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] The token cache is being invalidate because project 88f961f25de64533bf83beede20089cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:28.435618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.436402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-af417b7a-6a3e-4ce8-8a61-fc6cfe04a939 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:28.440751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69efc496-44e1-40fc-94d0-eac3e3c6dc7e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.444044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 893/1781] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:28 2022] DELETE /identity/v3/projects/88f961f25de64533bf83beede20089cf => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:28.445627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d0497c-b74e-42f8-802a-d2ca24ffa720 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.445918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d0497c-b74e-42f8-802a-d2ca24ffa720 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.446264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d0497c-b74e-42f8-802a-d2ca24ffa720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.472633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-41d0497c-b74e-42f8-802a-d2ca24ffa720 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:28.475146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-41d0497c-b74e-42f8-802a-d2ca24ffa720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:28.479321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:28.550810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.551560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.551963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.552405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.554361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.556908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.570732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 894/1782] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:28.579574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600cd872-1c4f-40e9-878b-a03e639d4f10 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.579883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600cd872-1c4f-40e9-878b-a03e639d4f10 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.583308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600cd872-1c4f-40e9-878b-a03e639d4f10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.583425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 889/1783] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/users => generated 296 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:28.584845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.586495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:28.599640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ac0a15-a607-4749-b05f-e4236d2fb760 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.604992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-600cd872-1c4f-40e9-878b-a03e639d4f10 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:28.605547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.606300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.606517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-600cd872-1c4f-40e9-878b-a03e639d4f10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:28.607130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.607410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.608106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.609769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.616378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 895/1784] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:28.621428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600cd872-1c4f-40e9-878b-a03e639d4f10 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.623107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 890/1785] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:28 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:28.626753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.633289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600cd872-1c4f-40e9-878b-a03e639d4f10 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.638725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.639359 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.639639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.639739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cb4f80-339e-4cf7-b2e0-2cbc6ef6fabe admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.639910 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.640611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:28.640909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.642485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.644656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.645337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.645668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.646004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.647032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.648802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.655736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 896/1786] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:28.660132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.669305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 891/1787] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:28 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:28.671109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f30c57d-def6-40f1-bba0-b80354f8ab5b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.673687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.677424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.678835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.679281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.679561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.680251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.682552 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.689841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b941a4-0755-4847-96ea-ee795fff3526 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.696936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84ce49a-577c-44be-bd94-d6dbbb66ce25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.697609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ce49a-577c-44be-bd94-d6dbbb66ce25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.697888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ce49a-577c-44be-bd94-d6dbbb66ce25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.698159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84ce49a-577c-44be-bd94-d6dbbb66ce25 admin admin] PATH_INFO: `/v3/projects/0c4c1688b9cc4606969402fc3e446a55/users/c4a13cc5e9344f4f8e5e89bafb02feb5/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.715756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84ce49a-577c-44be-bd94-d6dbbb66ce25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0c4c1688b9cc4606969402fc3e446a55, user_id=c4a13cc5e9344f4f8e5e89bafb02feb5, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.718551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84ce49a-577c-44be-bd94-d6dbbb66ce25 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.720088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 897/1788] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/projects => generated 375 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:54:28.722521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.734335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 892/1789] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:28 2022] PUT /identity/v3/projects/0c4c1688b9cc4606969402fc3e446a55/users/c4a13cc5e9344f4f8e5e89bafb02feb5/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:28.746294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f933111-842d-407c-adc2-d3e0131ed4cf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.746594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f933111-842d-407c-adc2-d3e0131ed4cf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.747303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f933111-842d-407c-adc2-d3e0131ed4cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.786508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5f933111-842d-407c-adc2-d3e0131ed4cf None None] MFA Rules not processed for user `c4a13cc5e9344f4f8e5e89bafb02feb5`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:28.788035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5f933111-842d-407c-adc2-d3e0131ed4cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:28.815299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc5f46d-f2bf-4a93-8907-7ce75971f81d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.856773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:28.872306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 893/1790] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:28.873591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f933111-842d-407c-adc2-d3e0131ed4cf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.887981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.888609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.888892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.889171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.889815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.891765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.917373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 898/1791] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/users => generated 288 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:28.921661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.933972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca4a64c-c075-4709-9168-389a00b35fb1 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.939326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:28.940167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.941766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.942510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.942895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.944401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.947348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.954961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f933111-842d-407c-adc2-d3e0131ed4cf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.961242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.961855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.962129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.962392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.963021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.964132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 899/1792] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:28 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:28.965054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:28.965595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:28.978311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95085f8d-506d-49ba-8f91-b78e1fced9e5 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:28.984429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:28.985107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:28.985398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:28.985672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:28.988178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:28.990151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.005574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 894/1793] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:28 2022] POST /identity/v3/projects => generated 393 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:29.005729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 900/1794] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:28 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:29.008540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.009266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.025409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d651ee-228a-4c5d-8ca6-131a6266b176 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:29.103566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:29.104370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.104648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.105456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] PATH_INFO: `/v3/projects/fa5c663186bc4b1d853726a52f70636e/users/a9620b4f6e0644f9ac7b4a543dc18578/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.110853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22561e3b-710f-4bc0-9173-8ef0f50ac828 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:29.117046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077009f0-1ae0-4630-af06-f5edebc08621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:29.118189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077009f0-1ae0-4630-af06-f5edebc08621 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.118621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077009f0-1ae0-4630-af06-f5edebc08621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.119049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077009f0-1ae0-4630-af06-f5edebc08621 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.119837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077009f0-1ae0-4630-af06-f5edebc08621 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:29.122590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077009f0-1ae0-4630-af06-f5edebc08621 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.124969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa5c663186bc4b1d853726a52f70636e, user_id=a9620b4f6e0644f9ac7b4a543dc18578, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:29.127599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.145560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 901/1795] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:29 2022] PUT /identity/v3/projects/fa5c663186bc4b1d853726a52f70636e/users/a9620b4f6e0644f9ac7b4a543dc18578/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:29.152262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 895/1796] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:29 2022] POST /identity/v3/users => generated 303 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:29.157082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.158824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.159116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.159386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.200856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] MFA Rules not processed for user `a9620b4f6e0644f9ac7b4a543dc18578`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:29.202097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:29.239701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8158b86-116d-492d-9891-b4b8c75e8fbe admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:29.247957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:29.248613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.248897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.249498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.250248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:29.252220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.266116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 902/1797] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:29 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:29.269996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.295129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f092f311-cef4-4ddb-ba17-8d846ce99296 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:29.302228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:29.303493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.303776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.304089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] PATH_INFO: `/v3/projects/04760eb5ac2747139176d815947244db/users/86e7898f1e904bcd9268b7d058a78589/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.348422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 896/1798] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:29.348758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=04760eb5ac2747139176d815947244db, user_id=86e7898f1e904bcd9268b7d058a78589, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:29.348758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.356281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:29.363517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.365993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:29.367078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 903/1799] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:29 2022] PUT /identity/v3/projects/04760eb5ac2747139176d815947244db/users/86e7898f1e904bcd9268b7d058a78589/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:29.377320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.462694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:29.492854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cd2056-825d-4bef-805c-be2d15cf02dc admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:29.498556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:29.499316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.499748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.500127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.500922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:29.503466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.503673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0497309-2ee3-432f-a09a-2c17cb09a5e8 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:29.530301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 904/1800] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:29 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:29.534557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.549554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d403499-5f14-4e97-9860-78f88614e59b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:29.567731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59358968-9520-4481-b7c1-9ff710d80086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:29.569081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59358968-9520-4481-b7c1-9ff710d80086 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.569437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59358968-9520-4481-b7c1-9ff710d80086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.569880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59358968-9520-4481-b7c1-9ff710d80086 admin admin] PATH_INFO: `/v3/projects/04760eb5ac2747139176d815947244db/users/86e7898f1e904bcd9268b7d058a78589/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.586981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59358968-9520-4481-b7c1-9ff710d80086 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04760eb5ac2747139176d815947244db, user_id=86e7898f1e904bcd9268b7d058a78589, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:29.591046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59358968-9520-4481-b7c1-9ff710d80086 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.612792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 905/1801] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:29 2022] PUT /identity/v3/projects/04760eb5ac2747139176d815947244db/users/86e7898f1e904bcd9268b7d058a78589/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:29.628798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:29.629666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.629666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.630524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.632574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ab4669-3bce-4a90-b502-8748645e487d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:29.633091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ab4669-3bce-4a90-b502-8748645e487d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:29.633723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ab4669-3bce-4a90-b502-8748645e487d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:29.697802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-78ab4669-3bce-4a90-b502-8748645e487d None None] MFA Rules not processed for user `86e7898f1e904bcd9268b7d058a78589`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:29.699264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-78ab4669-3bce-4a90-b502-8748645e487d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:29.720236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:29.722477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:29.739158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 897/1802] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:29.897815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 906/1803] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:29 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:29.909754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:29.911225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.019696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198412c9-8d71-4959-8b63-fb68ff4b21ed service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:30.039539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ab4669-3bce-4a90-b502-8748645e487d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:30.072361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:30.080597 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e6ba77-c581-4788-ad6d-f06c74e89684 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:30.081294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e6ba77-c581-4788-ad6d-f06c74e89684 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.082010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e6ba77-c581-4788-ad6d-f06c74e89684 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.082010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e6ba77-c581-4788-ad6d-f06c74e89684 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.097543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e6ba77-c581-4788-ad6d-f06c74e89684 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:30.100384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e6ba77-c581-4788-ad6d-f06c74e89684 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:30.111747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 898/1804] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:29 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:30.113036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c21fa6-4dc8-4bbd-a5aa-510d3c305a7e None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.113321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c21fa6-4dc8-4bbd-a5aa-510d3c305a7e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.113590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c21fa6-4dc8-4bbd-a5aa-510d3c305a7e None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.114805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 899/1805] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:54:30 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:54:30.121027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ab4669-3bce-4a90-b502-8748645e487d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:30.121417 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.121704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.121977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.126691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:30.127333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.127616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.127887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] PATH_INFO: `/v3/users/a9620b4f6e0644f9ac7b4a543dc18578` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.129666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9620b4f6e0644f9ac7b4a543dc18578)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:30.133045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:30.140923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:30.145043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:30.146251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.198773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9620b4f6e0644f9ac7b4a543dc18578'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:30.199473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9620b4f6e0644f9ac7b4a543dc18578'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:30.200523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9620b4f6e0644f9ac7b4a543dc18578'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:30.207795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9620b4f6e0644f9ac7b4a543dc18578'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:30.220349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9620b4f6e0644f9ac7b4a543dc18578'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:30.229196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 900/1806] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:54:30 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:30.230092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:30.231532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.243809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eb789c-1f54-4174-9a3b-cc1341000556 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:30.244018 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 907/1807] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:30 2022] DELETE /identity/v3/users/a9620b4f6e0644f9ac7b4a543dc18578 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:30.246916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:30.246916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.307569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:30.308213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.308497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.308761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.311159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.325327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dab160-7638-4a25-81eb-e3b5e714d252 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:30.330858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60968fb0-2659-414a-a254-1aebe8e9f0e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:30.331471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60968fb0-2659-414a-a254-1aebe8e9f0e0 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.331743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60968fb0-2659-414a-a254-1aebe8e9f0e0 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.332055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60968fb0-2659-414a-a254-1aebe8e9f0e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.341111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60968fb0-2659-414a-a254-1aebe8e9f0e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:30.342699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60968fb0-2659-414a-a254-1aebe8e9f0e0 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:30.359238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 908/1808] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:30.362708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.362999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.364268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.385251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:30.386739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:30.399225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:30.400721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.416684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 901/1809] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:30 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:30.489678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 909/1810] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:30.500728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:30.502857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.565638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:30.589950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a6e683-2abf-4379-8c62-2f8a70849782 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:30.598105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3071cf-f095-41f7-893b-bc28af992305 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:30.598956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3071cf-f095-41f7-893b-bc28af992305 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:30.599298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3071cf-f095-41f7-893b-bc28af992305 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:30.599766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3071cf-f095-41f7-893b-bc28af992305 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:30.613079 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3071cf-f095-41f7-893b-bc28af992305 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:30.614736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3071cf-f095-41f7-893b-bc28af992305 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:30.625835 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 902/1811] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:30.975049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:30.976816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:30.993143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7616c096-e21c-4547-9745-ef06c38ff1e9 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:30.999184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.000064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.000309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.000652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] PATH_INFO: `/v3/projects/fa5c663186bc4b1d853726a52f70636e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.011097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa5c663186bc4b1d853726a52f70636e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.013178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.019887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:31.063379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa5c663186bc4b1d853726a52f70636e', 'request_id': 'req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:31.070232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa5c663186bc4b1d853726a52f70636e', 'request_id': 'req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:31.075859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa5c663186bc4b1d853726a52f70636e', 'request_id': 'req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:31.089454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] The token cache is being invalidate because project fa5c663186bc4b1d853726a52f70636e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:31.089797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:31.090561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-0dc3927b-99c4-4e2c-82f3-a0ab47c233ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:31.092503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 910/1812] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:30 2022] DELETE /identity/v3/projects/fa5c663186bc4b1d853726a52f70636e => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:31.097435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b2594-2333-45e6-a0fb-b8ba9859958a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.097867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b2594-2333-45e6-a0fb-b8ba9859958a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.097984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9b2594-2333-45e6-a0fb-b8ba9859958a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.119067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7f9b2594-2333-45e6-a0fb-b8ba9859958a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:31.120223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7f9b2594-2333-45e6-a0fb-b8ba9859958a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:31.187139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 903/1813] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:31.197808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af59954-3487-4154-90df-71c09cb708e6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.198094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af59954-3487-4154-90df-71c09cb708e6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.198366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af59954-3487-4154-90df-71c09cb708e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.219453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6af59954-3487-4154-90df-71c09cb708e6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:31.220768 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6af59954-3487-4154-90df-71c09cb708e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:31.229550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 911/1814] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:31.233724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b2594-2333-45e6-a0fb-b8ba9859958a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.243376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9b2594-2333-45e6-a0fb-b8ba9859958a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.248106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.248784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.249073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.249340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.249976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:31.250270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.251853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.268305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 904/1815] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:31.269502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af59954-3487-4154-90df-71c09cb708e6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.281334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af59954-3487-4154-90df-71c09cb708e6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.284631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.285225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.285499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.285761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.286411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.288923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.323562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 912/1816] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:31 2022] POST /identity/v3/projects => generated 369 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:31.324982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.340347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98c1aa0-62c9-4cc4-9c9c-1cd77fac2be3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.402888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.403491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.403771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.404189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.404830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.406734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.433747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 905/1817] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:31 2022] POST /identity/v3/users => generated 285 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:31.434975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.446376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361f63e1-cf55-414d-8222-2c3ad0be486c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.450982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.452621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.452898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.453164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.453912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.455477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.474362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 913/1818] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:31 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:31.479756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.491488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab1ff1a-7498-4507-9a5a-8bc8bfb00571 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.496995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994075e3-5db8-4854-a2eb-7dbac667c036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.497600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994075e3-5db8-4854-a2eb-7dbac667c036 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.497879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994075e3-5db8-4854-a2eb-7dbac667c036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.498155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994075e3-5db8-4854-a2eb-7dbac667c036 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.499213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994075e3-5db8-4854-a2eb-7dbac667c036 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.501169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994075e3-5db8-4854-a2eb-7dbac667c036 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.516621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 906/1819] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:31 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:31.522871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.535078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9013abc9-c29b-40d7-b3ea-23b704365c6b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.541593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.542318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.542611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.542877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] PATH_INFO: `/v3/projects/145bfcddb2b043549052a56ff6a17f9f/users/55ed9d56b6334e51b9488033112b4f77/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.561809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=145bfcddb2b043549052a56ff6a17f9f, user_id=55ed9d56b6334e51b9488033112b4f77, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.564522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.577880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 914/1820] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:31 2022] PUT /identity/v3/projects/145bfcddb2b043549052a56ff6a17f9f/users/55ed9d56b6334e51b9488033112b4f77/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:31.588602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295de446-a52a-49e9-a3d3-f0257297d3da None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.588895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295de446-a52a-49e9-a3d3-f0257297d3da None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.589172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295de446-a52a-49e9-a3d3-f0257297d3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.622978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-295de446-a52a-49e9-a3d3-f0257297d3da None None] MFA Rules not processed for user `55ed9d56b6334e51b9488033112b4f77`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:31.624242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-295de446-a52a-49e9-a3d3-f0257297d3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:31.664831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:31.699549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 907/1821] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:31 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:31.704204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.769571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5bc65f-8337-4892-a270-530476f2b2ef admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.774425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.775011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.775291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.775560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.776272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.778167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.811341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 915/1822] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:31 2022] POST /identity/v3/projects => generated 369 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:54:31.812940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295de446-a52a-49e9-a3d3-f0257297d3da None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.824353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295de446-a52a-49e9-a3d3-f0257297d3da None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.882873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.883464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.883738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.884024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.884643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.886639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.908145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 908/1823] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:31 2022] POST /identity/v3/users => generated 291 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:31.912869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.923367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc23526-e9ce-4f32-9d7c-20d5badd9335 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.928802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.929440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.929722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.929991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.930636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.932266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.943328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 916/1824] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:31.947893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:31.958125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06ee18a-3716-40af-8856-3627875e0d44 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:31.962794 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:31.963452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:31.963732 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:31.964027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] PATH_INFO: `/v3/projects/a3dbcc0192744c589012b432a109fada/users/44c9f8bb80ed49b1a47905fc0dc78218/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:31.979297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3dbcc0192744c589012b432a109fada, user_id=44c9f8bb80ed49b1a47905fc0dc78218, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:31.981966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:31.996105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 909/1825] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:31 2022] PUT /identity/v3/projects/a3dbcc0192744c589012b432a109fada/users/44c9f8bb80ed49b1a47905fc0dc78218/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:32.000152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:32.008244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:32.012351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c051e6fe-e2d8-4eee-b73c-70b8612a43aa admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:32.071782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf02d31-13ef-4fdc-b64d-6d654ce7a065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:32.072431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf02d31-13ef-4fdc-b64d-6d654ce7a065 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:32.072706 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf02d31-13ef-4fdc-b64d-6d654ce7a065 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:32.072972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf02d31-13ef-4fdc-b64d-6d654ce7a065 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:32.073637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf02d31-13ef-4fdc-b64d-6d654ce7a065 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:32.075202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf02d31-13ef-4fdc-b64d-6d654ce7a065 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:32.086326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 917/1826] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:31 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:32.089601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:32.101823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f58606-7e87-45b1-87b4-9e321918f24e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:32.107856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:32.108530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:32.108804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:32.109071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] PATH_INFO: `/v3/projects/a3dbcc0192744c589012b432a109fada/users/44c9f8bb80ed49b1a47905fc0dc78218/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:32.110978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3dbcc0192744c589012b432a109fada, user_id=44c9f8bb80ed49b1a47905fc0dc78218, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:32.113642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:32.124830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 910/1827] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:32 2022] PUT /identity/v3/projects/a3dbcc0192744c589012b432a109fada/users/44c9f8bb80ed49b1a47905fc0dc78218/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:32.137135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:32.137447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:32.137718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:32.168778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] MFA Rules not processed for user `44c9f8bb80ed49b1a47905fc0dc78218`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:32.170005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:32.254465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 918/1828] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:32 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:32.261859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:32.263677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:32.330412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6805e66f-5a7e-4ac2-8fcc-96636ed13a95 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:32.334825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786d76b4-64be-4ab9-b81e-671d6063daa9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:32.335425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786d76b4-64be-4ab9-b81e-671d6063daa9 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:32.335699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786d76b4-64be-4ab9-b81e-671d6063daa9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:32.335986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786d76b4-64be-4ab9-b81e-671d6063daa9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:32.345463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786d76b4-64be-4ab9-b81e-671d6063daa9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:32.346986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786d76b4-64be-4ab9-b81e-671d6063daa9 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:32.357333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 911/1829] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:32 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:34.227084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:34.229056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:34.408477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c830b89-8337-4f13-a0d4-b07e37870400 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:34.457376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:34.458046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:34.458328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:34.462222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] PATH_INFO: `/v3/users/552197ac405a44ca8ae03201cd2a1245` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:34.464747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=552197ac405a44ca8ae03201cd2a1245)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:34.478512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:34.566167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:34.598488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '552197ac405a44ca8ae03201cd2a1245'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:34.600246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '552197ac405a44ca8ae03201cd2a1245'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:34.601025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '552197ac405a44ca8ae03201cd2a1245'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:34.614794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '552197ac405a44ca8ae03201cd2a1245'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:34.633126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '552197ac405a44ca8ae03201cd2a1245'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:34.650425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 919/1830] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:34 2022] DELETE /identity/v3/users/552197ac405a44ca8ae03201cd2a1245 => generated 0 bytes in 424 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:34.655133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:34.655626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:34.655769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:34.695068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:34.696921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:34.717911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:34.720041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:34.748679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:34.858022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 912/1831] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:34.862593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:34.865775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:34.879164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeef20a-783d-4fcf-9976-99116a88ad5d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:34.890796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:34.891831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:34.894104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:34.894518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] PATH_INFO: `/v3/users/55ed9d56b6334e51b9488033112b4f77` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:34.898924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] RBAC: Authorizing `identity:delete_user(user_id=55ed9d56b6334e51b9488033112b4f77)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:34.904556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:35.035752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55ed9d56b6334e51b9488033112b4f77'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.036185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89ad0d8-bbf0-4ccf-b5c2-53a696e41fb5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:35.037632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55ed9d56b6334e51b9488033112b4f77'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.038435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55ed9d56b6334e51b9488033112b4f77'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.047218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284c3196-5c65-47b9-8c8a-b0a5362edc40 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:35.047850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284c3196-5c65-47b9-8c8a-b0a5362edc40 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:35.048156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284c3196-5c65-47b9-8c8a-b0a5362edc40 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:35.048427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284c3196-5c65-47b9-8c8a-b0a5362edc40 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:35.071105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-284c3196-5c65-47b9-8c8a-b0a5362edc40 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:35.074399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55ed9d56b6334e51b9488033112b4f77'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.099154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284c3196-5c65-47b9-8c8a-b0a5362edc40 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:35.103882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284c3196-5c65-47b9-8c8a-b0a5362edc40 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:35.104584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55ed9d56b6334e51b9488033112b4f77'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.146008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 913/1832] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:35.146298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 920/1833] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:34 2022] DELETE /identity/v3/users/55ed9d56b6334e51b9488033112b4f77 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:35.150803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:35.151090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:35.151363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:35.197412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:35.198778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:35.322358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 914/1834] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:35.344691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:35.347485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:35.385127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:35.430181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-babe4f96-e151-4661-9b1c-4b9791203a1b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:35.450762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d3b28-e590-4b24-9784-54b9819298e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:35.452612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d3b28-e590-4b24-9784-54b9819298e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:35.453799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d3b28-e590-4b24-9784-54b9819298e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:35.454349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d3b28-e590-4b24-9784-54b9819298e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:35.479728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d3b28-e590-4b24-9784-54b9819298e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:35.484009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d3b28-e590-4b24-9784-54b9819298e5 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:35.508801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 921/1835] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:35.671828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:35.677364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:35.723527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c559a6-d5dc-4228-a0f4-39d472dbf22e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:35.777251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:35.783849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:35.784424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:35.784846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] PATH_INFO: `/v3/users/c4a13cc5e9344f4f8e5e89bafb02feb5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:35.791196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4a13cc5e9344f4f8e5e89bafb02feb5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:35.806637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:35.933292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4a13cc5e9344f4f8e5e89bafb02feb5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.934801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4a13cc5e9344f4f8e5e89bafb02feb5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.968783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4a13cc5e9344f4f8e5e89bafb02feb5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:35.970218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4a13cc5e9344f4f8e5e89bafb02feb5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:36.003122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:36.014396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4a13cc5e9344f4f8e5e89bafb02feb5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:36.046839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 915/1836] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:35 2022] DELETE /identity/v3/users/c4a13cc5e9344f4f8e5e89bafb02feb5 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:36.050977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:36.051271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:36.051543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:36.087715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:36.091261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:36.200261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 922/1837] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:36.232728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:36.234238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:36.360898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-834fb5d2-10c1-4613-b2bc-9e071407d766 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:36.364688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:36.365970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:36.366247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:36.366512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:36.376906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:36.378348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:36.389882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 916/1838] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:36.758732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:36.760932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:36.783528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:36.785063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:36.805932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baddae8f-5c26-4061-9dc1-3544e6cc8590 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:36.814731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d557d0-4701-4146-aef6-df9531ca8125 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:36.826616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:36.827383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:36.829742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:36.832357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:36.832605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:36.832923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] PATH_INFO: `/v3/projects/f43b9cc8d197448faab988d13724667f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:36.834233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:36.835230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f43b9cc8d197448faab988d13724667f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:36.835582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] PATH_INFO: `/v3/projects/145bfcddb2b043549052a56ff6a17f9f` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:36.837296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:36.838256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] RBAC: Authorizing `identity:delete_project(project_id=145bfcddb2b043549052a56ff6a17f9f)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:36.842290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:36.951757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '145bfcddb2b043549052a56ff6a17f9f', 'request_id': 'req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:36.955290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f43b9cc8d197448faab988d13724667f', 'request_id': 'req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:36.968637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '145bfcddb2b043549052a56ff6a17f9f', 'request_id': 'req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:36.975991 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '145bfcddb2b043549052a56ff6a17f9f', 'request_id': 'req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:36.980636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f43b9cc8d197448faab988d13724667f', 'request_id': 'req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:36.993548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f43b9cc8d197448faab988d13724667f', 'request_id': 'req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.001670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] The token cache is being invalidate because project 145bfcddb2b043549052a56ff6a17f9f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:37.002044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.002923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.005716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 917/1839] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:36 2022] DELETE /identity/v3/projects/145bfcddb2b043549052a56ff6a17f9f => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:37.010766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:37.012910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:37.018100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] The token cache is being invalidate because project f43b9cc8d197448faab988d13724667f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:37.018379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.019181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.024431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 923/1840] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:36 2022] DELETE /identity/v3/projects/f43b9cc8d197448faab988d13724667f => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:37.030045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:37.033032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:37.124443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:37.170343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fad57a-11a5-4c28-bea2-2ec0fb9ce813 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:37.174233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff8d289-fcb7-4b06-9a93-77dd09a37ec3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:37.182948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:37.183921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:37.184402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:37.184839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] PATH_INFO: `/v3/users/44c9f8bb80ed49b1a47905fc0dc78218` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:37.186605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=44c9f8bb80ed49b1a47905fc0dc78218)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:37.189258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:37.202716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:37.203385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:37.203691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:37.210722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] PATH_INFO: `/v3/users/28eb794b7f264d2db783a898d0db9e2d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:37.213542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28eb794b7f264d2db783a898d0db9e2d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:37.215542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:37.219636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:37.291824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44c9f8bb80ed49b1a47905fc0dc78218'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.292708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44c9f8bb80ed49b1a47905fc0dc78218'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.304846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44c9f8bb80ed49b1a47905fc0dc78218'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.306075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44c9f8bb80ed49b1a47905fc0dc78218'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.313532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28eb794b7f264d2db783a898d0db9e2d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.315974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28eb794b7f264d2db783a898d0db9e2d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.315974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28eb794b7f264d2db783a898d0db9e2d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.324510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28eb794b7f264d2db783a898d0db9e2d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.330038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44c9f8bb80ed49b1a47905fc0dc78218'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.345923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a67a420-f508-4d3f-acd6-2b345b056a53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28eb794b7f264d2db783a898d0db9e2d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.352592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 918/1841] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:37 2022] DELETE /identity/v3/users/44c9f8bb80ed49b1a47905fc0dc78218 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:37.356207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:37.358107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:37.367048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 924/1842] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:37 2022] DELETE /identity/v3/users/28eb794b7f264d2db783a898d0db9e2d => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:37.457427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cba1e5b-2af7-40df-91dd-24a5a8f7ac0e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:37.467397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d15ccbe-b1c9-4a5b-b212-404b4928970f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:37.467648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d15ccbe-b1c9-4a5b-b212-404b4928970f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:37.467879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d15ccbe-b1c9-4a5b-b212-404b4928970f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:37.469403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 925/1843] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:54:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:54:37.494566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:37.494892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:37.495175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:37.544446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:37.549206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:37.612199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:37.614307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:37.615419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:37.616166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] PATH_INFO: `/v3/projects/0c4c1688b9cc4606969402fc3e446a55` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:37.619488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=0c4c1688b9cc4606969402fc3e446a55)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:37.628142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:37.712870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 926/1844] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:54:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:37.730425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:37.731327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:37.732199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:37.745145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0c4c1688b9cc4606969402fc3e446a55', 'request_id': 'req-af12b641-e85d-487f-8c8e-f09c5c7a13cb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.756947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0c4c1688b9cc4606969402fc3e446a55', 'request_id': 'req-af12b641-e85d-487f-8c8e-f09c5c7a13cb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.763149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb903bfa-f29e-4bac-9ace-9b38e315d28c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:37.765443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0c4c1688b9cc4606969402fc3e446a55', 'request_id': 'req-af12b641-e85d-487f-8c8e-f09c5c7a13cb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.780808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] The token cache is being invalidate because project 0c4c1688b9cc4606969402fc3e446a55 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:37.781279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.782158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:37.783879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 919/1845] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:37 2022] DELETE /identity/v3/projects/0c4c1688b9cc4606969402fc3e446a55 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:37.788323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:37.790030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:37.887065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:37.911236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:37.913429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:37.915182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:37.915495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:37.919350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:37.921677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af12b641-e85d-487f-8c8e-f09c5c7a13cb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:37.926491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:37.927891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:37.928292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:37.928749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] PATH_INFO: `/v3/users/86e7898f1e904bcd9268b7d058a78589` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:37.930154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86e7898f1e904bcd9268b7d058a78589)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:37.932209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:37.951187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:37.953065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:37.965816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 927/1846] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:38.017530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86e7898f1e904bcd9268b7d058a78589'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.018564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86e7898f1e904bcd9268b7d058a78589'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.033350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86e7898f1e904bcd9268b7d058a78589'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.034656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86e7898f1e904bcd9268b7d058a78589'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.071528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86e7898f1e904bcd9268b7d058a78589'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.109225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 920/1847] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:37 2022] DELETE /identity/v3/users/86e7898f1e904bcd9268b7d058a78589 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:38.303430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.304955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:38.320939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.322706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:38.341575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ca7ed2-54f0-4ba8-9d2f-2086850fe9dd service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.437490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:38.442938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:38.447981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.447981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.447981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] PATH_INFO: `/v3/projects/a3dbcc0192744c589012b432a109fada` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.449018 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3dbcc0192744c589012b432a109fada)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:38.451569 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:38.466540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac72a9d-a5f5-4f74-beda-79130b679d10 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.477385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:38.478516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.479406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.479887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] PATH_INFO: `/v3/projects/ffb50e3d06bb41daab42057d2dfc0f05` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.481994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffb50e3d06bb41daab42057d2dfc0f05)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:38.485037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:38.512898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3dbcc0192744c589012b432a109fada', 'request_id': 'req-e78ffa74-e0ae-4691-9ec9-87a581877e28'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.520581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3dbcc0192744c589012b432a109fada', 'request_id': 'req-e78ffa74-e0ae-4691-9ec9-87a581877e28'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.529106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3dbcc0192744c589012b432a109fada', 'request_id': 'req-e78ffa74-e0ae-4691-9ec9-87a581877e28'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.544698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffb50e3d06bb41daab42057d2dfc0f05', 'request_id': 'req-fdd5fceb-9730-4105-90cb-4458a89d127c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.545148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] The token cache is being invalidate because project a3dbcc0192744c589012b432a109fada was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:38.545449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.546106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e78ffa74-e0ae-4691-9ec9-87a581877e28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.547619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 928/1848] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:38 2022] DELETE /identity/v3/projects/a3dbcc0192744c589012b432a109fada => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:38.553523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ef59d-c17c-473a-8d80-53d15bd18429 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.553927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ef59d-c17c-473a-8d80-53d15bd18429 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.554293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ef59d-c17c-473a-8d80-53d15bd18429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.557978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffb50e3d06bb41daab42057d2dfc0f05', 'request_id': 'req-fdd5fceb-9730-4105-90cb-4458a89d127c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.566766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffb50e3d06bb41daab42057d2dfc0f05', 'request_id': 'req-fdd5fceb-9730-4105-90cb-4458a89d127c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.584774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] The token cache is being invalidate because project ffb50e3d06bb41daab42057d2dfc0f05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:38.585086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.585779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fdd5fceb-9730-4105-90cb-4458a89d127c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:38.587173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5d9ef59d-c17c-473a-8d80-53d15bd18429 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:38.588739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5d9ef59d-c17c-473a-8d80-53d15bd18429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:38.590901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 921/1849] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:38 2022] DELETE /identity/v3/projects/ffb50e3d06bb41daab42057d2dfc0f05 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:38.597268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b806387-772d-4d69-93b2-8bc05a07cc0f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.598240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b806387-772d-4d69-93b2-8bc05a07cc0f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.598240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b806387-772d-4d69-93b2-8bc05a07cc0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.623370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8b806387-772d-4d69-93b2-8bc05a07cc0f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:38.624734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8b806387-772d-4d69-93b2-8bc05a07cc0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:38.672038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 929/1850] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:38.680967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d63f2-45f4-4670-81c3-ca14e5ec4c40 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.681349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d63f2-45f4-4670-81c3-ca14e5ec4c40 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.681692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583d63f2-45f4-4670-81c3-ca14e5ec4c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.703034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-583d63f2-45f4-4670-81c3-ca14e5ec4c40 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:38.704248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-583d63f2-45f4-4670-81c3-ca14e5ec4c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:38.705311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 922/1851] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:38.714587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9778d5-720b-4553-ad69-d519dedd16d9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.714962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9778d5-720b-4553-ad69-d519dedd16d9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.715448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9778d5-720b-4553-ad69-d519dedd16d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.720156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 930/1852] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:38.723826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d63f2-45f4-4670-81c3-ca14e5ec4c40 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.737263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583d63f2-45f4-4670-81c3-ca14e5ec4c40 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.743281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ac9778d5-720b-4553-ad69-d519dedd16d9 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:38.744772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ac9778d5-720b-4553-ad69-d519dedd16d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:38.744941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:38.745816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.746290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.746734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.747835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:38.748385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:38.755137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:38.763272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:38.765348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 923/1853] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:38.767140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9778d5-720b-4553-ad69-d519dedd16d9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.780565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 931/1854] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:54:38.782814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.786097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9778d5-720b-4553-ad69-d519dedd16d9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.792957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:38.793571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.793843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.794106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.794744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:38.795057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723b4f52-74bf-44f6-ab90-641ce59bc22b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.795437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:38.796972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:38.811406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:38.812541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.812925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 924/1855] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:38.813214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.813490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.814734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:38.818578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:38.819614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.830376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fecad6c-a807-4261-bd69-afd7c53d7c0e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.838458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:38.839066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.839459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.839987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.841401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:38.844402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:38.866861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 932/1856] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:38 2022] POST /identity/v3/projects => generated 367 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:38.868579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.871326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:38.887379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 925/1857] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:38 2022] POST /identity/v3/projects => generated 379 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:38.889013 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:38.946015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:38.984994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21aa9f33-2770-45d7-92d9-59fd94bfb696 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.992093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aa15c0-8219-4fe3-bec1-4d732f57bdb9 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:38.994507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:38.995670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:38.996256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:38.996896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:38.997950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:38.999662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.000345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.000625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.000946 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] PATH_INFO: `/v3/projects/04760eb5ac2747139176d815947244db` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.002230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] RBAC: Authorizing `identity:delete_project(project_id=04760eb5ac2747139176d815947244db)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.002350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.004180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.045325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 926/1858] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:38 2022] POST /identity/v3/users => generated 284 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:39.055270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.087141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cef92aa-36c3-410f-a321-d8d0ab8f3d49 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.099574 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04760eb5ac2747139176d815947244db', 'request_id': 'req-8fa9dc3d-ff5e-4449-998d-41f841cc032c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:39.110471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04760eb5ac2747139176d815947244db', 'request_id': 'req-8fa9dc3d-ff5e-4449-998d-41f841cc032c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:39.120823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04760eb5ac2747139176d815947244db', 'request_id': 'req-8fa9dc3d-ff5e-4449-998d-41f841cc032c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:39.140194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] The token cache is being invalidate because project 04760eb5ac2747139176d815947244db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:39.140634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:39.142424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:39.144842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 933/1859] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:38 2022] DELETE /identity/v3/projects/04760eb5ac2747139176d815947244db => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:39.146222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.147900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:39.205980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3c6473-f453-4634-a77a-208127758ceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.206591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c6473-f453-4634-a77a-208127758ceb admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.206875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c6473-f453-4634-a77a-208127758ceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.207145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3c6473-f453-4634-a77a-208127758ceb admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.209017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3c6473-f453-4634-a77a-208127758ceb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.211999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3c6473-f453-4634-a77a-208127758ceb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.267150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 927/1860] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:39 2022] POST /identity/v3/users => generated 290 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:39.269744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.272790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.274547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.307913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa9dc3d-ff5e-4449-998d-41f841cc032c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.312967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:39.314353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:39.316611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16ec62e-25cb-4a1a-839c-10316db05053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.317400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16ec62e-25cb-4a1a-839c-10316db05053 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.318010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16ec62e-25cb-4a1a-839c-10316db05053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.318423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16ec62e-25cb-4a1a-839c-10316db05053 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.319819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16ec62e-25cb-4a1a-839c-10316db05053 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.324377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 928/1861] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:39.324377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.326952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:39.329674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16ec62e-25cb-4a1a-839c-10316db05053 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.344759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c4d5c7-272f-4e3d-b280-d751d27bae29 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.348305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 934/1862] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:39.351044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b9f5e-a896-4d1f-841c-8c86cb0086ac None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.351541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b9f5e-a896-4d1f-841c-8c86cb0086ac None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.352339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71b9f5e-a896-4d1f-841c-8c86cb0086ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.355186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.356401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.357065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.357594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.358973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.362328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.363279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:39.383050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 929/1863] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:39.384433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.395497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a71b9f5e-a896-4d1f-841c-8c86cb0086ac None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:39.398086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a71b9f5e-a896-4d1f-841c-8c86cb0086ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:39.403792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3961ef65-865a-4913-9597-4b8f5af86887 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.409421 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 935/1864] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:39.410633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71b9f5e-a896-4d1f-841c-8c86cb0086ac None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.414977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.415829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.416421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.416943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.417824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.419700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.429077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71b9f5e-a896-4d1f-841c-8c86cb0086ac None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.437342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.438133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.438874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.439616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.440460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.446266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 930/1865] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:39.448057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.462185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.484826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 936/1866] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:39 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:39.488666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc6664f-bb56-4cc8-8871-2b5bcb9814ca admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.492064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.510891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.511922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.513733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.513733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.518194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:39.519112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.522299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.536302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63babaab-c2cc-4d17-af70-ea8a6db6a367 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.550725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.551709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.551963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.553120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] PATH_INFO: `/v3/projects/39c82c2591fd46e9b01773a3e51a9901/users/79c5f3c9109749599c4a035baefb49c1/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.564220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 931/1867] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:39.564220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.589396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec610cc7-9ecc-40a9-97f7-bf322da0ac93 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.590939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=39c82c2591fd46e9b01773a3e51a9901, user_id=79c5f3c9109749599c4a035baefb49c1, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.594905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.604540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a8c023-41ca-47c3-b63b-15e796316c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.605554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8c023-41ca-47c3-b63b-15e796316c81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.605979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8c023-41ca-47c3-b63b-15e796316c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.606393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8c023-41ca-47c3-b63b-15e796316c81 admin admin] PATH_INFO: `/v3/projects/b5036e542c9b4660bae75ec91a8a3f8b/users/ecf13a6144b54ae6b12601331b99a307/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.633996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a8c023-41ca-47c3-b63b-15e796316c81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5036e542c9b4660bae75ec91a8a3f8b, user_id=ecf13a6144b54ae6b12601331b99a307, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.639530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a8c023-41ca-47c3-b63b-15e796316c81 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.643160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 937/1868] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:39 2022] PUT /identity/v3/projects/39c82c2591fd46e9b01773a3e51a9901/users/79c5f3c9109749599c4a035baefb49c1/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:39.644595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:39.681856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 932/1869] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:39 2022] PUT /identity/v3/projects/b5036e542c9b4660bae75ec91a8a3f8b/users/ecf13a6144b54ae6b12601331b99a307/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:39.687081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b531ee-0762-47d5-9856-50898c57c90d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.687387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b531ee-0762-47d5-9856-50898c57c90d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.687661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b531ee-0762-47d5-9856-50898c57c90d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.773934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc14123-cdd3-4065-a4b5-85767e29ef8c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:39.808252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b8b531ee-0762-47d5-9856-50898c57c90d None None] MFA Rules not processed for user `79c5f3c9109749599c4a035baefb49c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:39.810407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b8b531ee-0762-47d5-9856-50898c57c90d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:39.892508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:39.901486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:39.902108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.902390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.905917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.907595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:39.914944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:39.941449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 933/1870] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:39 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:39.943453 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c7c39-935b-4a1b-af07-0816b5dd426f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:39.943756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c7c39-935b-4a1b-af07-0816b5dd426f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:39.944058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5c7c39-935b-4a1b-af07-0816b5dd426f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:39.997973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1d5c7c39-935b-4a1b-af07-0816b5dd426f None None] MFA Rules not processed for user `ecf13a6144b54ae6b12601331b99a307`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:39.999300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1d5c7c39-935b-4a1b-af07-0816b5dd426f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:40.024506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 938/1871] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:39 2022] POST /identity/v3/projects => generated 381 bytes in 378 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:40.024506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.118028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:40.121455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 934/1872] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:39 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:40.127509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5c7c39-935b-4a1b-af07-0816b5dd426f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.141808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6705c6-0be6-42dc-a7d6-e9fda503f3a1 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.147493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.148220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.148709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.149157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.149994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.152451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5c7c39-935b-4a1b-af07-0816b5dd426f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.153323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.163510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.164811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.165206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.165546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.166315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.172544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.190969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 939/1873] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/projects => generated 369 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:40.190969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.202050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 935/1874] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/users => generated 291 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:40.203264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.216993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f213d6-9e0c-4db4-b4cb-8b73ee7a6779 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.264312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7576b5b-7260-46a5-8f91-cdfde261b6ac admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.269006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.269633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.269908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.270173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.270790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.274290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.292415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.293026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.293303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.293571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.294203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.296428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.305915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 940/1875] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/projects => generated 379 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:40.306950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.325634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 936/1876] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/users => generated 291 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:40.325634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.335884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e124ed-d6a4-4f6c-b15c-58678018b2e2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.371429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03faff87-dec0-46ff-a3f5-639b54504329 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.375279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:40.376117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.376736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.376992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.377303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.377989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.380021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.391282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 941/1877] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:40.392490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.393551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.394406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.394690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.394962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.395584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.398607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.403317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c433766d-444d-4011-8353-e04aa253fcec admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.407924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.408532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.408806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.409073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.409700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.411261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.420615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 937/1878] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/users => generated 290 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:40.421544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.423011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 942/1879] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:40.424592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.433917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13209836-d37c-4c31-8752-1113859c3be6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.435310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3502956-9b15-406b-949f-313be0e74ffd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.438768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.439369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.439701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.439895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.440130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.440507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.440785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.440912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.441068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.441671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.442373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.443878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.454037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 943/1880] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:40.457264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 938/1881] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:40.457390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.457976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.466846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7d21c-3edf-44c6-b3d8-07ad0cf94a8f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.468220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0cae078-9259-4d05-98ec-2cdd9d757b04 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.473187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.473689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.474076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.474622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.474731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.474885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] PATH_INFO: `/v3/projects/3fe31f9d15b445e1b25e325491ca214c/users/31ea30c437c44fb08435656d7030509d/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.475075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.475075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.475865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.477770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.493262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 939/1882] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:40.493262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.493491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fe31f9d15b445e1b25e325491ca214c, user_id=31ea30c437c44fb08435656d7030509d, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.495840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.502766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff466b0-82f9-4698-a60a-8daf7bff2e73 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.509345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.510188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.510566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.511215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] PATH_INFO: `/v3/projects/7a5e20f9185e4f0bb95264dfb484e56c/users/8fb7b16bf988482d846278bb0cda1238/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.515097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 944/1883] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:40 2022] PUT /identity/v3/projects/3fe31f9d15b445e1b25e325491ca214c/users/31ea30c437c44fb08435656d7030509d/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:40.515097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.530308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a5e20f9185e4f0bb95264dfb484e56c, user_id=8fb7b16bf988482d846278bb0cda1238, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.532908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:40.533250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.550915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 940/1884] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:40 2022] PUT /identity/v3/projects/7a5e20f9185e4f0bb95264dfb484e56c/users/8fb7b16bf988482d846278bb0cda1238/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:40.552768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.600546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eac7a0c-e2cd-4da0-bdfe-9af5c17bc368 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.634938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c80bf499-44d6-4604-8b43-f8ba312f2aa8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.640308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381cc275-38aa-4aaa-b17a-07aa0383f6db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.641141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381cc275-38aa-4aaa-b17a-07aa0383f6db admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.641515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381cc275-38aa-4aaa-b17a-07aa0383f6db admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.642075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381cc275-38aa-4aaa-b17a-07aa0383f6db admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.643955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381cc275-38aa-4aaa-b17a-07aa0383f6db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.646247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381cc275-38aa-4aaa-b17a-07aa0383f6db admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.662684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 941/1885] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:40 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:40.664269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91100385-8f20-4b8b-8a5a-037b471f11bf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.664680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.665512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.666049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.666165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] PATH_INFO: `/v3/projects/72f058280223479fb7853896b7fd7394/users/a85e1d0358234634b5c4f19dbd8b2836/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.668293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91100385-8f20-4b8b-8a5a-037b471f11bf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.668293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91100385-8f20-4b8b-8a5a-037b471f11bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.683571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=72f058280223479fb7853896b7fd7394, user_id=a85e1d0358234634b5c4f19dbd8b2836, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.686347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.704470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 945/1886] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:40 2022] PUT /identity/v3/projects/72f058280223479fb7853896b7fd7394/users/a85e1d0358234634b5c4f19dbd8b2836/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:40.705337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.708212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-91100385-8f20-4b8b-8a5a-037b471f11bf None None] MFA Rules not processed for user `8fb7b16bf988482d846278bb0cda1238`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:40.710161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-91100385-8f20-4b8b-8a5a-037b471f11bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:40.778844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f53068-95e9-4d12-90f0-7beea53a10ce admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.784284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 942/1887] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:40.785680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c953204-1ce3-46b3-9996-0b012052f6e8 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.786040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c953204-1ce3-46b3-9996-0b012052f6e8 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.786382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c953204-1ce3-46b3-9996-0b012052f6e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.789379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.790059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.790340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.790738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] PATH_INFO: `/v3/projects/3fe31f9d15b445e1b25e325491ca214c/users/31ea30c437c44fb08435656d7030509d/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.792716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fe31f9d15b445e1b25e325491ca214c, user_id=31ea30c437c44fb08435656d7030509d, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.796582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.812779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 946/1888] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:40 2022] PUT /identity/v3/projects/3fe31f9d15b445e1b25e325491ca214c/users/31ea30c437c44fb08435656d7030509d/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:40.813213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.822860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:40.827474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6c953204-1ce3-46b3-9996-0b012052f6e8 None None] MFA Rules not processed for user `a85e1d0358234634b5c4f19dbd8b2836`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:40.829357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6c953204-1ce3-46b3-9996-0b012052f6e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:40.868165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:40.896120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae7f426-c22e-4ed0-827a-ff0bd8dfce6b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:40.903904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:40.904538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.904816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.905084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.905707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:40.907674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:40.916844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 943/1889] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:40.918569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb39319-fbb0-4f83-aaa8-620ef314ad90 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:40.918855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb39319-fbb0-4f83-aaa8-620ef314ad90 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:40.919126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb39319-fbb0-4f83-aaa8-620ef314ad90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:40.935997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 947/1890] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/projects => generated 383 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:40.936962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:40.948319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6bb39319-fbb0-4f83-aaa8-620ef314ad90 None None] MFA Rules not processed for user `31ea30c437c44fb08435656d7030509d`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:40.949523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6bb39319-fbb0-4f83-aaa8-620ef314ad90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:41.000113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da02f857-4a5d-4d1b-a72d-fe35335ecebb admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.005725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.006390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.006660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.007040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.007667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.009637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.019320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 944/1891] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:41.020176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb39319-fbb0-4f83-aaa8-620ef314ad90 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.032047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb39319-fbb0-4f83-aaa8-620ef314ad90 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.039347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.040280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.040559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.040824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.041476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.043372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.049523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 948/1892] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:40 2022] POST /identity/v3/projects => generated 379 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:41.049523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.051479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:41.070887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 945/1893] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:41 2022] POST /identity/v3/users => generated 298 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:41.070887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.080766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9d1942-f68a-4be9-a28c-47270e06c760 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.121188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09139d2b-f9dd-4cc6-88ff-c448dabaf104 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.125580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.126181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.126455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.126721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.146359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.147822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.148227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.148580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.149608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.152183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.173084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:41.179902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 946/1894] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:41 2022] POST /identity/v3/users => generated 296 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:41.180819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.195540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1066ea-d0fe-415c-9e36-1d9cbc1855f0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.212165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.212165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.212165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.212443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.212531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.212531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.212690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.212771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.212851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:41.220233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 949/1895] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:41.220586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 947/1896] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:41.221458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.225012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.234313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1314e7a-ff1c-4a23-9e22-5b6824a5a17b service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.236393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95630f99-7449-4687-935b-269116a44f22 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.238900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.239718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.240048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.240319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.241088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.241816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.242828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.242949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.243180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.243565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] PATH_INFO: `/v3/projects/0379b9b287ee4854adb4ef9660e7fd8a/users/3a87977879e648799fbb335ee86b8c3e/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.254240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 950/1897] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:41.265337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.276293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829661c7-c4da-4312-8eb4-c1cf0760385f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.278385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0379b9b287ee4854adb4ef9660e7fd8a, user_id=3a87977879e648799fbb335ee86b8c3e, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.281070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.281208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.281721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.282102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.282259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] PATH_INFO: `/v3/projects/500b5a975ae64af08e4443cd284ad02a/users/33698dbcf4b04d168e8eda15a30c495f/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.296729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 948/1898] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:41 2022] PUT /identity/v3/projects/0379b9b287ee4854adb4ef9660e7fd8a/users/3a87977879e648799fbb335ee86b8c3e/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:41.301079 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.301203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] RBAC: Authorizing `identity:create_grant(project_id=500b5a975ae64af08e4443cd284ad02a, user_id=33698dbcf4b04d168e8eda15a30c495f, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.304033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:41.304303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.318919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 951/1899] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:41 2022] PUT /identity/v3/projects/500b5a975ae64af08e4443cd284ad02a/users/33698dbcf4b04d168e8eda15a30c495f/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:41.322977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.369999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ff31fd-c5a7-4056-b28a-ef1d4f07571a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.392367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a11fac-4316-41c9-8e73-38f49a167218 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.398150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.398845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.399198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.399472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.400283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.401943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.413119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 952/1900] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:41.414016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.425689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdb4ac-2174-486c-9528-89160d4f084b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.426108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70068f1c-257a-41d4-b856-d965bf3efb54 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.428081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70068f1c-257a-41d4-b856-d965bf3efb54 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.429285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70068f1c-257a-41d4-b856-d965bf3efb54 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.429659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70068f1c-257a-41d4-b856-d965bf3efb54 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.431971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.432615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.432897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.433259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.434031 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.435648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.449152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 953/1901] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:41.449152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.456884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd1239-3961-4454-8509-00d83d48c482 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.462120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.462794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.463073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.463345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] PATH_INFO: `/v3/projects/0379b9b287ee4854adb4ef9660e7fd8a/users/3a87977879e648799fbb335ee86b8c3e/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.465281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=0379b9b287ee4854adb4ef9660e7fd8a, user_id=3a87977879e648799fbb335ee86b8c3e, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.467868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.481013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 954/1902] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:41 2022] PUT /identity/v3/projects/0379b9b287ee4854adb4ef9660e7fd8a/users/3a87977879e648799fbb335ee86b8c3e/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:41.485322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.497254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70068f1c-257a-41d4-b856-d965bf3efb54 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.498682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70068f1c-257a-41d4-b856-d965bf3efb54 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.508054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 949/1903] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 211 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:41.509110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f00db-e539-49ea-b8ca-9acc3ac9af1d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.509392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f00db-e539-49ea-b8ca-9acc3ac9af1d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.509659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231f00db-e539-49ea-b8ca-9acc3ac9af1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.539639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-231f00db-e539-49ea-b8ca-9acc3ac9af1d None None] MFA Rules not processed for user `3a87977879e648799fbb335ee86b8c3e`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:41.540832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-231f00db-e539-49ea-b8ca-9acc3ac9af1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:41.554496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fef48a-b214-40e3-a3ef-a26542bf75dc admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.559740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3225be-b9a1-484e-a9f3-d41090302938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.560429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3225be-b9a1-484e-a9f3-d41090302938 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.560744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3225be-b9a1-484e-a9f3-d41090302938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.561015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3225be-b9a1-484e-a9f3-d41090302938 admin admin] PATH_INFO: `/v3/projects/500b5a975ae64af08e4443cd284ad02a/users/33698dbcf4b04d168e8eda15a30c495f/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.562939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3225be-b9a1-484e-a9f3-d41090302938 admin admin] RBAC: Authorizing `identity:create_grant(project_id=500b5a975ae64af08e4443cd284ad02a, user_id=33698dbcf4b04d168e8eda15a30c495f, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.565688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3225be-b9a1-484e-a9f3-d41090302938 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.572617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:41.578451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 955/1904] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:41 2022] PUT /identity/v3/projects/500b5a975ae64af08e4443cd284ad02a/users/33698dbcf4b04d168e8eda15a30c495f/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:41.589027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7170be68-5032-4935-9da8-24738982d041 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.589513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7170be68-5032-4935-9da8-24738982d041 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.589795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7170be68-5032-4935-9da8-24738982d041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.621508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7170be68-5032-4935-9da8-24738982d041 None None] MFA Rules not processed for user `33698dbcf4b04d168e8eda15a30c495f`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:41.623214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7170be68-5032-4935-9da8-24738982d041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:41.631772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 950/1905] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:41 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:41.643337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231f00db-e539-49ea-b8ca-9acc3ac9af1d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.649044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:41.723209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231f00db-e539-49ea-b8ca-9acc3ac9af1d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.723421 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 956/1906] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:41 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:41.784634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.785259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.785544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.785815 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.871904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.873408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.885277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 951/1907] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:41.908883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7170be68-5032-4935-9da8-24738982d041 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:41.920104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7170be68-5032-4935-9da8-24738982d041 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:41.924222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:41.926489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:41.926939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:41.927258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:41.964109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:41.965559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:41.977153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 957/1908] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:41 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:42.025758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:42.036070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570999f3-ec7c-403a-b4c9-0e967c7d20a2 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:42.042789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:42.043978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:42.044296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:42.044622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:42.073211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:42.076728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:42.087802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4e82b8-1726-415a-9888-9d78681877aa service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:42.091980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0ea905-5688-4461-b969-523cb56b1c23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:42.092758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0ea905-5688-4461-b969-523cb56b1c23 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:42.093252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0ea905-5688-4461-b969-523cb56b1c23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:42.093526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0ea905-5688-4461-b969-523cb56b1c23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:42.112874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:42.114310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:42.126628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 952/1909] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:42 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:42.162980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0ea905-5688-4461-b969-523cb56b1c23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:42.164471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0ea905-5688-4461-b969-523cb56b1c23 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:42.173586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 958/1910] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:42 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:45.106894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:45.168599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc846cb-f122-4d71-8e35-8943c6a2d753 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:45.173084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:45.173702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:45.173981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:45.174253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] PATH_INFO: `/v3/users/79c5f3c9109749599c4a035baefb49c1` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:45.175376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=79c5f3c9109749599c4a035baefb49c1)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:45.177487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:45.228773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79c5f3c9109749599c4a035baefb49c1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.229413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79c5f3c9109749599c4a035baefb49c1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.236743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79c5f3c9109749599c4a035baefb49c1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.237383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79c5f3c9109749599c4a035baefb49c1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.249965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79c5f3c9109749599c4a035baefb49c1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.263876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 953/1911] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:45 2022] DELETE /identity/v3/users/79c5f3c9109749599c4a035baefb49c1 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:45.268637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:45.269064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:45.269425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:45.290750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:45.292152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:45.356024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 959/1912] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:45.362745 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:45.364172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:45.412710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:45.424680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea664d7d-d163-4f62-ab88-a80336f8e3bb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:45.428679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:45.429307 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:45.429586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:45.429854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:45.438829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:45.440293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:45.450308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 954/1913] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:45.781039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:45.782888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:45.796143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd13b85-84ca-4b48-904f-27e7a6321ddd None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:45.814842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:45.815691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:45.816230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:45.816635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] PATH_INFO: `/v3/projects/39c82c2591fd46e9b01773a3e51a9901` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:45.818927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] RBAC: Authorizing `identity:delete_project(project_id=39c82c2591fd46e9b01773a3e51a9901)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:45.821548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:45.880689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:45.899774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39c82c2591fd46e9b01773a3e51a9901', 'request_id': 'req-25b2ee5f-8154-4c47-998a-5eb9aac8150c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.907573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39c82c2591fd46e9b01773a3e51a9901', 'request_id': 'req-25b2ee5f-8154-4c47-998a-5eb9aac8150c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.917629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39c82c2591fd46e9b01773a3e51a9901', 'request_id': 'req-25b2ee5f-8154-4c47-998a-5eb9aac8150c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.933874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] The token cache is being invalidate because project 39c82c2591fd46e9b01773a3e51a9901 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:45.934491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.935485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:45.937277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 960/1914] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:45 2022] DELETE /identity/v3/projects/39c82c2591fd46e9b01773a3e51a9901 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:45.941603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:45.953138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a69ab50-6dcb-47b8-82a1-60a6bd08bb88 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:46.043496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:46.044176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.044455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.044734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] PATH_INFO: `/v3/users/31ea30c437c44fb08435656d7030509d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.046499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31ea30c437c44fb08435656d7030509d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:46.048609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:46.109701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31ea30c437c44fb08435656d7030509d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.110727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31ea30c437c44fb08435656d7030509d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.118196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31ea30c437c44fb08435656d7030509d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.118842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31ea30c437c44fb08435656d7030509d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.131594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5fa95e55-44ff-4726-a2ce-53d0fdc03e61 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31ea30c437c44fb08435656d7030509d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.152586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 955/1915] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:45 2022] DELETE /identity/v3/users/31ea30c437c44fb08435656d7030509d => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:46.417636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:46.419067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:46.432634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b2ee5f-8154-4c47-998a-5eb9aac8150c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:46.514529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:46.515158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.515437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.515701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] PATH_INFO: `/v3/projects/3fe31f9d15b445e1b25e325491ca214c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.516858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fe31f9d15b445e1b25e325491ca214c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:46.518772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:46.569181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fe31f9d15b445e1b25e325491ca214c', 'request_id': 'req-abf27735-bd5f-4795-a9b1-9180851b3958'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.575286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fe31f9d15b445e1b25e325491ca214c', 'request_id': 'req-abf27735-bd5f-4795-a9b1-9180851b3958'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.580937 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fe31f9d15b445e1b25e325491ca214c', 'request_id': 'req-abf27735-bd5f-4795-a9b1-9180851b3958'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.594891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] The token cache is being invalidate because project 3fe31f9d15b445e1b25e325491ca214c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:46.595179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.596028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abf27735-bd5f-4795-a9b1-9180851b3958 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:46.598049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 961/1916] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:46 2022] DELETE /identity/v3/projects/3fe31f9d15b445e1b25e325491ca214c => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:46.603038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd94312-4061-4fbc-923b-18fc917d20fd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.603329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd94312-4061-4fbc-923b-18fc917d20fd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.603598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd94312-4061-4fbc-923b-18fc917d20fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.624720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-cfd94312-4061-4fbc-923b-18fc917d20fd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:46.625908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cfd94312-4061-4fbc-923b-18fc917d20fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:46.684873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 956/1917] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:46.693261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433c3de-7bab-4bc2-8685-43afcaa744ab None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.693756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433c3de-7bab-4bc2-8685-43afcaa744ab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.694096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5433c3de-7bab-4bc2-8685-43afcaa744ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.714237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5433c3de-7bab-4bc2-8685-43afcaa744ab None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:46.715269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5433c3de-7bab-4bc2-8685-43afcaa744ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:46.723562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 962/1918] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:46.728130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd94312-4061-4fbc-923b-18fc917d20fd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:46.734220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:46.738414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd94312-4061-4fbc-923b-18fc917d20fd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:46.745105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:46.745717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.746221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.746720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.747581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:46.748040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:46.749669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:46.762652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 957/1919] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:46.766928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5433c3de-7bab-4bc2-8685-43afcaa744ab None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:46.776309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5433c3de-7bab-4bc2-8685-43afcaa744ab None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:46.780888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:46.781479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.781754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.782019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.782649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:46.784598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:46.822779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 963/1920] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:46 2022] POST /identity/v3/projects => generated 361 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:46.826935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:46.837400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0f7c5f-559d-4106-a725-223e8d6aaa70 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:46.897496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:46.898100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.898374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.898638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.899266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:46.901276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:46.924106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 958/1921] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:46 2022] POST /identity/v3/users => generated 281 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:46.928581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:46.938812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ff5964-5387-4530-bf2c-4024a8d63020 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:46.944384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:46.944974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.945248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.945511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.946136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:46.947702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:46.957774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 964/1922] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:46.961835 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:46.971920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd0b659-45fb-4265-b625-0f860a252a90 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:46.977365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a8fcb5-6d3a-409e-b3a8-69fd8248517a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:46.977975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8fcb5-6d3a-409e-b3a8-69fd8248517a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:46.978252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8fcb5-6d3a-409e-b3a8-69fd8248517a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:46.978517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8fcb5-6d3a-409e-b3a8-69fd8248517a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:46.979202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a8fcb5-6d3a-409e-b3a8-69fd8248517a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:46.980818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a8fcb5-6d3a-409e-b3a8-69fd8248517a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:46.991883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 959/1923] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:46.995998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:47.006357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b70a2-2aa7-4b17-989f-a91aba042d14 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:47.010988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:47.011629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:47.011912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:47.012219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] PATH_INFO: `/v3/projects/fa08ef223aea41f8a9a32d17fd658522/users/1504be341ab5476e90e06f35ff9a4dad/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:47.023243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:47.027704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa08ef223aea41f8a9a32d17fd658522, user_id=1504be341ab5476e90e06f35ff9a4dad, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:47.030659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:47.044853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 965/1924] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:46 2022] PUT /identity/v3/projects/fa08ef223aea41f8a9a32d17fd658522/users/1504be341ab5476e90e06f35ff9a4dad/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:47.056797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4815b25-343d-4029-8168-d18e722a5bfe None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:47.057112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4815b25-343d-4029-8168-d18e722a5bfe None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:47.057381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4815b25-343d-4029-8168-d18e722a5bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:47.092529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e4815b25-343d-4029-8168-d18e722a5bfe None None] MFA Rules not processed for user `1504be341ab5476e90e06f35ff9a4dad`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:47.092529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e4815b25-343d-4029-8168-d18e722a5bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:47.158760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 960/1925] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:47 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:47.195179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:47.196988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:47.264144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14fb2eb-3718-4891-be59-8320c0c13a97 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:47.268224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31720daa-b6d3-4985-8118-c6bcc0d6eae7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:47.268916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31720daa-b6d3-4985-8118-c6bcc0d6eae7 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:47.269259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31720daa-b6d3-4985-8118-c6bcc0d6eae7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:47.269589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31720daa-b6d3-4985-8118-c6bcc0d6eae7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:47.280096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31720daa-b6d3-4985-8118-c6bcc0d6eae7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:47.281515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31720daa-b6d3-4985-8118-c6bcc0d6eae7 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:47.291178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 966/1926] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:47 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:47.306993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4815b25-343d-4029-8168-d18e722a5bfe None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:47.374221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4815b25-343d-4029-8168-d18e722a5bfe None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:47.383514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:47.384181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:47.384444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:47.384712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] PATH_INFO: `/v3/users/1504be341ab5476e90e06f35ff9a4dad` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:47.385850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=1504be341ab5476e90e06f35ff9a4dad)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:47.387762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:47.389369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:47.434810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1504be341ab5476e90e06f35ff9a4dad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:47.435448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1504be341ab5476e90e06f35ff9a4dad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:47.442483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1504be341ab5476e90e06f35ff9a4dad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:47.443081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1504be341ab5476e90e06f35ff9a4dad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:47.454802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1504be341ab5476e90e06f35ff9a4dad'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:47.467887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 961/1927] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:47 2022] DELETE /identity/v3/users/1504be341ab5476e90e06f35ff9a4dad => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:47.472394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:47.472863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:47.472982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:47.495915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:47.497136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:47.557332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 967/1928] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:47.567251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:47.568971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:47.634893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae4c75e-b3be-4994-81c2-3d1707f68cfb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:47.638518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d508e6d-e740-4cbd-b7af-5f21d108a173 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:47.639114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d508e6d-e740-4cbd-b7af-5f21d108a173 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:47.639389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d508e6d-e740-4cbd-b7af-5f21d108a173 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:47.639657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d508e6d-e740-4cbd-b7af-5f21d108a173 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:47.650045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d508e6d-e740-4cbd-b7af-5f21d108a173 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:47.651471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d508e6d-e740-4cbd-b7af-5f21d108a173 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:47.661233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 962/1929] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:47.932233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:47.933914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:47.948440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2d60d-2971-460a-a398-88af83655756 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:47.953372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:47.954011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:47.954290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:47.954560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] PATH_INFO: `/v3/projects/fa08ef223aea41f8a9a32d17fd658522` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:47.955919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa08ef223aea41f8a9a32d17fd658522)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:47.957754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.014978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa08ef223aea41f8a9a32d17fd658522', 'request_id': 'req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:48.022269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa08ef223aea41f8a9a32d17fd658522', 'request_id': 'req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:48.028709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa08ef223aea41f8a9a32d17fd658522', 'request_id': 'req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:48.043210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] The token cache is being invalidate because project fa08ef223aea41f8a9a32d17fd658522 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:48.043501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:48.044171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8f4ec4bb-3751-466a-ae84-1b18339ce1e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:48.045659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 968/1930] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:47 2022] DELETE /identity/v3/projects/fa08ef223aea41f8a9a32d17fd658522 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:48.050589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa736171-6be5-4e5d-8d5a-40cc06fc5112 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.051030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa736171-6be5-4e5d-8d5a-40cc06fc5112 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.051379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa736171-6be5-4e5d-8d5a-40cc06fc5112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.071864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fa736171-6be5-4e5d-8d5a-40cc06fc5112 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:48.073092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fa736171-6be5-4e5d-8d5a-40cc06fc5112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:48.134420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 963/1931] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:48.142553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb7cc3d-87b2-4608-8550-eafe14518054 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.142845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb7cc3d-87b2-4608-8550-eafe14518054 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.143125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb7cc3d-87b2-4608-8550-eafe14518054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.163208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-cfb7cc3d-87b2-4608-8550-eafe14518054 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:48.164436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cfb7cc3d-87b2-4608-8550-eafe14518054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:48.172260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 969/1932] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:48.176407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa736171-6be5-4e5d-8d5a-40cc06fc5112 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.186776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa736171-6be5-4e5d-8d5a-40cc06fc5112 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.191309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.191887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.192187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.193056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.193927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:48.194224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.195800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.207072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 964/1933] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:48.211169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb7cc3d-87b2-4608-8550-eafe14518054 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.220340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb7cc3d-87b2-4608-8550-eafe14518054 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.225164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.225758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.226038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.226310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.226933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.228890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.234278 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:48.259579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 970/1934] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:48.263588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.274205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed030b4e-5366-4496-8fb0-55950153b1df admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.278738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:48.333922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.334522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.334801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.335070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.335689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.337635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.359017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 965/1935] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:48.363298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.374354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c754c970-619a-433a-904c-21e62617b360 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.379236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.380961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.381423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.381698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.382374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.384031 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.396197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 971/1936] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:48.400131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.424138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41956e07-8cb8-4406-aa30-ee3ca2c3c438 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.430545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf042ef-4f4d-47c6-b068-724f66610270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.431137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf042ef-4f4d-47c6-b068-724f66610270 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.431465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf042ef-4f4d-47c6-b068-724f66610270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.431742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf042ef-4f4d-47c6-b068-724f66610270 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.432506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf042ef-4f4d-47c6-b068-724f66610270 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.434225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf042ef-4f4d-47c6-b068-724f66610270 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.447052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 966/1937] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:48 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:48.451626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.462078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d56b7f-c613-40f0-a3e4-f5425755d235 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.466669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.467316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.467615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.467873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] PATH_INFO: `/v3/projects/91c9fd811c6848cca297e03cf0da9642/users/2f7707a279104f248f1c16db2f6cfde7/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.484871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91c9fd811c6848cca297e03cf0da9642, user_id=2f7707a279104f248f1c16db2f6cfde7, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.488521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.503521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 972/1938] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:48 2022] PUT /identity/v3/projects/91c9fd811c6848cca297e03cf0da9642/users/2f7707a279104f248f1c16db2f6cfde7/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:48.514907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ca420-0f30-4dbf-b08f-1b19343bca75 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.515193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ca420-0f30-4dbf-b08f-1b19343bca75 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.515467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ca420-0f30-4dbf-b08f-1b19343bca75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.545403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fb9ca420-0f30-4dbf-b08f-1b19343bca75 None None] MFA Rules not processed for user `2f7707a279104f248f1c16db2f6cfde7`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:48.546729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fb9ca420-0f30-4dbf-b08f-1b19343bca75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:48.611440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 967/1939] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:48.615988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.680237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e93d88-cea9-4eae-963c-03010e7644e9 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.685007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.685608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.685888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.686158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.686791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.688730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.716127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 973/1940] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/projects => generated 385 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:48.719534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9ca420-0f30-4dbf-b08f-1b19343bca75 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.730059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9ca420-0f30-4dbf-b08f-1b19343bca75 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.791460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.792264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.792567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.792858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.793604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.795598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.819741 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 968/1941] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:48.823887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.834493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d05621-1c28-49d4-a711-09f35505db5c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.839076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.839739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.840061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.840375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.841039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.842646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.853925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 974/1942] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:48.858050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.868965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe330722-fd95-4101-a6d3-0f9f52e93c76 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.873626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b2c350-f0a9-4f30-b8e4-3e2c36453eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.874368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b2c350-f0a9-4f30-b8e4-3e2c36453eba admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.874648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b2c350-f0a9-4f30-b8e4-3e2c36453eba admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.874916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b2c350-f0a9-4f30-b8e4-3e2c36453eba admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.875559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b2c350-f0a9-4f30-b8e4-3e2c36453eba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.877215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b2c350-f0a9-4f30-b8e4-3e2c36453eba admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.878576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:48.890672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 969/1943] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:48.894686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:48.905126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd05f0-b44a-485b-9425-10a86b695e7c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:48.909935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:48.910591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.910871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.911202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] PATH_INFO: `/v3/projects/64315132b3ab4ec28f514445ba2270e7/users/19105be8be244c1791367ebb616f38f9/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.926618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64315132b3ab4ec28f514445ba2270e7, user_id=19105be8be244c1791367ebb616f38f9, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:48.929902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:48.943681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 975/1944] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:48 2022] PUT /identity/v3/projects/64315132b3ab4ec28f514445ba2270e7/users/19105be8be244c1791367ebb616f38f9/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:48.954974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5dcf94-ecb0-417c-a1a3-cac0b2d3085e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:48.955287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5dcf94-ecb0-417c-a1a3-cac0b2d3085e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:48.955539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5dcf94-ecb0-417c-a1a3-cac0b2d3085e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:48.983475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4c5dcf94-ecb0-417c-a1a3-cac0b2d3085e None None] MFA Rules not processed for user `19105be8be244c1791367ebb616f38f9`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:48.984625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4c5dcf94-ecb0-417c-a1a3-cac0b2d3085e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:49.049241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 970/1945] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:49.055067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:49.123707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e7f669-668d-4f72-a346-a63394f166a4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:49.128762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:49.129371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.129924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.130819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.131489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:49.133432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:49.138964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:49.164107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 976/1946] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:49 2022] POST /identity/v3/projects => generated 385 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:49.167151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5dcf94-ecb0-417c-a1a3-cac0b2d3085e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:49.180540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5dcf94-ecb0-417c-a1a3-cac0b2d3085e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:49.258138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:49.258748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.259050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.259319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.259964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:49.261908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:49.283360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 971/1947] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:49 2022] POST /identity/v3/users => generated 299 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:54:49.287718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:49.297990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65abb5c0-9aef-448f-9767-98ee97a5c254 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:49.302731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:49.303339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.303616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.303883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.304558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:49.306140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:49.315859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 977/1948] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:49.320508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:49.330298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbc2793-1715-47cc-905d-8baf9bbfe6c8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:49.335769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:49.336455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.336751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.337044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] PATH_INFO: `/v3/projects/e95a767470864a4296f963b2769727b5/users/c864836a410d411bb929e6a311bc5558/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.351450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e95a767470864a4296f963b2769727b5, user_id=c864836a410d411bb929e6a311bc5558, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:49.354117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:49.367011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 972/1949] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:49 2022] PUT /identity/v3/projects/e95a767470864a4296f963b2769727b5/users/c864836a410d411bb929e6a311bc5558/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:49.371034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:49.382187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45c385-ed30-4795-8ad8-04ddbe84f925 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:49.437838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2394e3-e498-4537-9be4-3193d0c1efab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:49.438483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2394e3-e498-4537-9be4-3193d0c1efab admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.438771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2394e3-e498-4537-9be4-3193d0c1efab admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.439045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2394e3-e498-4537-9be4-3193d0c1efab admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.439691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2394e3-e498-4537-9be4-3193d0c1efab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:49.441313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2394e3-e498-4537-9be4-3193d0c1efab admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:49.452336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 978/1950] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:49 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:49.456536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:49.472592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef193489-8e09-4e10-8199-e9e9db3f564b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:49.478790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:49.479495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.479770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.480114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] PATH_INFO: `/v3/projects/e95a767470864a4296f963b2769727b5/users/c864836a410d411bb929e6a311bc5558/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.482219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=e95a767470864a4296f963b2769727b5, user_id=c864836a410d411bb929e6a311bc5558, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:49.484932 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:49.496110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 973/1951] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:49 2022] PUT /identity/v3/projects/e95a767470864a4296f963b2769727b5/users/c864836a410d411bb929e6a311bc5558/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:49.507067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.507566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.507652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.537056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] MFA Rules not processed for user `c864836a410d411bb929e6a311bc5558`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:49.538289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:49.615276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 979/1952] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:49 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:49.627600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:49.629546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:49.688445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:49.709701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae78e9b-0e4a-46af-b208-22f8d7b168cf admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:49.715002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:49.715799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:49.716129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:49.716402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:49.802811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:49.804402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:49.815488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 974/1953] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:51.368539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:51.371471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:51.480507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:51.485418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a42a52-940f-47b2-a24e-e86283c834e6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:51.491980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:51.492788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:51.493252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:51.493522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] PATH_INFO: `/v3/users/8fb7b16bf988482d846278bb0cda1238` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:51.494654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fb7b16bf988482d846278bb0cda1238)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:51.496735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:51.557221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fb7b16bf988482d846278bb0cda1238'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.558140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fb7b16bf988482d846278bb0cda1238'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.559431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fb7b16bf988482d846278bb0cda1238'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.571194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:51.572716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:51.573983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fb7b16bf988482d846278bb0cda1238'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.585436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9e1887-77d2-436c-b3eb-c772fe9cdf41 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:51.590057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:51.590612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:51.591318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:51.591596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:51.593406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:51.605633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-283fc55f-1678-4885-a263-237d14cd4464 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fb7b16bf988482d846278bb0cda1238'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.635064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 980/1954] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:51 2022] DELETE /identity/v3/users/8fb7b16bf988482d846278bb0cda1238 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:51.640172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:51.640461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:51.640736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:51.664695 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:51.665916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:51.670201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:51.671925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:51.683631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 975/1955] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:51 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:51.743491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 981/1956] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:51.752126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:51.753641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:51.826871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:51.827455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:51.832394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496c58cb-9643-483e-9007-2ba0037384ad service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:51.842951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67ce601-d7e5-4064-b01a-13a0caf07ce1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:51.849283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:51.849695 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:51.849975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:51.850245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] PATH_INFO: `/v3/users/ecf13a6144b54ae6b12601331b99a307` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:51.851983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecf13a6144b54ae6b12601331b99a307)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:51.853944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:51.907012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:51.916465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007c22f7-1452-4a44-9e51-5d5f8d811bc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:51.916957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c22f7-1452-4a44-9e51-5d5f8d811bc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:51.917240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c22f7-1452-4a44-9e51-5d5f8d811bc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:51.917940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007c22f7-1452-4a44-9e51-5d5f8d811bc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:51.930968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007c22f7-1452-4a44-9e51-5d5f8d811bc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:51.933716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007c22f7-1452-4a44-9e51-5d5f8d811bc6 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:51.934238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecf13a6144b54ae6b12601331b99a307'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.934944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecf13a6144b54ae6b12601331b99a307'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.935666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecf13a6144b54ae6b12601331b99a307'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.951368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 976/1957] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:51.954970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecf13a6144b54ae6b12601331b99a307'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:51.969637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:51.973124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecf13a6144b54ae6b12601331b99a307'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.011874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 982/1958] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:51 2022] DELETE /identity/v3/users/ecf13a6144b54ae6b12601331b99a307 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:52.017019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:52.017595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:52.018193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:52.044598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:52.046310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:52.126654 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 977/1959] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:52.155531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:52.157155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:52.249640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060e2012-a80e-4109-ae79-cbf81a0bd771 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:52.253378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:52.253982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:52.254259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:52.254573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:52.264907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:52.266338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:52.279710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 983/1960] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:52.477521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:52.479078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:52.507316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5162c381-45bf-40f9-ad09-a7b0dd1e3ffc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:52.521260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:52.521900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:52.522178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:52.522456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] PATH_INFO: `/v3/projects/7a5e20f9185e4f0bb95264dfb484e56c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:52.523647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a5e20f9185e4f0bb95264dfb484e56c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:52.525673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:52.604681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a5e20f9185e4f0bb95264dfb484e56c', 'request_id': 'req-14287166-7e97-4b3a-95d5-cf7fd2869d02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.615357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a5e20f9185e4f0bb95264dfb484e56c', 'request_id': 'req-14287166-7e97-4b3a-95d5-cf7fd2869d02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.623124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a5e20f9185e4f0bb95264dfb484e56c', 'request_id': 'req-14287166-7e97-4b3a-95d5-cf7fd2869d02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.652130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] The token cache is being invalidate because project 7a5e20f9185e4f0bb95264dfb484e56c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:52.652430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.653016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:52.653505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.655635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 978/1961] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:52 2022] DELETE /identity/v3/projects/7a5e20f9185e4f0bb95264dfb484e56c => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:52.656658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:52.660762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:52.665110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:52.785268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14287166-7e97-4b3a-95d5-cf7fd2869d02 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:52.793319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:52.793964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:52.794240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:52.794510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] PATH_INFO: `/v3/users/3a87977879e648799fbb335ee86b8c3e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:52.795658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a87977879e648799fbb335ee86b8c3e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:52.800014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:52.804169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8ec7e2-f4e7-478a-a50e-b046124b840f service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:52.817149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:52.874774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:52.875804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:52.876525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:52.877001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:52.880431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:52.888236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a87977879e648799fbb335ee86b8c3e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.889489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a87977879e648799fbb335ee86b8c3e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.900764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a87977879e648799fbb335ee86b8c3e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.901519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a87977879e648799fbb335ee86b8c3e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.924476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a87977879e648799fbb335ee86b8c3e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:52.945647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 979/1962] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:52 2022] DELETE /identity/v3/users/3a87977879e648799fbb335ee86b8c3e => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:52.974377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:52.976403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:52.982095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:53.014109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 984/1963] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:52 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:54:53.224775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:53.227864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:53.358793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b57e02-0339-4bb5-a2f0-96e320ebe0d5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:53.363710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:53.364962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:53.365242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:53.365512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] PATH_INFO: `/v3/projects/b5036e542c9b4660bae75ec91a8a3f8b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:53.366663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5036e542c9b4660bae75ec91a8a3f8b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:53.368639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:53.426401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5036e542c9b4660bae75ec91a8a3f8b', 'request_id': 'req-68876e6f-0483-45ed-89cb-e6a6d840111d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.444696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5036e542c9b4660bae75ec91a8a3f8b', 'request_id': 'req-68876e6f-0483-45ed-89cb-e6a6d840111d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.451739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5036e542c9b4660bae75ec91a8a3f8b', 'request_id': 'req-68876e6f-0483-45ed-89cb-e6a6d840111d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.468373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] The token cache is being invalidate because project b5036e542c9b4660bae75ec91a8a3f8b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:53.469042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.469825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.471423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 980/1964] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:53 2022] DELETE /identity/v3/projects/b5036e542c9b4660bae75ec91a8a3f8b => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:53.475823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:53.480966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:53.511800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c99e5a-c264-457a-9fe9-85b260c429c4 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:53.578720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:53.579369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:53.579705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:53.580096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] PATH_INFO: `/v3/users/a85e1d0358234634b5c4f19dbd8b2836` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:53.581283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a85e1d0358234634b5c4f19dbd8b2836)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:53.583632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:53.633332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a85e1d0358234634b5c4f19dbd8b2836'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.634090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a85e1d0358234634b5c4f19dbd8b2836'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.634754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a85e1d0358234634b5c4f19dbd8b2836'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.649543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a85e1d0358234634b5c4f19dbd8b2836'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.669388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a4452c4e-4387-49d7-b139-443bde46ed9a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a85e1d0358234634b5c4f19dbd8b2836'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.684130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 985/1965] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:53 2022] DELETE /identity/v3/users/a85e1d0358234634b5c4f19dbd8b2836 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:53.702969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:53.704698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:53.720566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68876e6f-0483-45ed-89cb-e6a6d840111d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:53.757843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:53.862877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:53.863529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:53.863821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:53.864211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] PATH_INFO: `/v3/projects/0379b9b287ee4854adb4ef9660e7fd8a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:53.866143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0379b9b287ee4854adb4ef9660e7fd8a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:53.868081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:53.938525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0379b9b287ee4854adb4ef9660e7fd8a', 'request_id': 'req-599a72e3-506a-443d-93fb-582a6b16b43e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.951070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0379b9b287ee4854adb4ef9660e7fd8a', 'request_id': 'req-599a72e3-506a-443d-93fb-582a6b16b43e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.959802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0379b9b287ee4854adb4ef9660e7fd8a', 'request_id': 'req-599a72e3-506a-443d-93fb-582a6b16b43e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.985175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] The token cache is being invalidate because project 0379b9b287ee4854adb4ef9660e7fd8a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:53.989601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.990676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-599a72e3-506a-443d-93fb-582a6b16b43e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:53.996750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 981/1966] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:53 2022] DELETE /identity/v3/projects/0379b9b287ee4854adb4ef9660e7fd8a => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:53.998724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:53.999018 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:53.999735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.038896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:54.046593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:54.187265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 986/1967] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:54.203100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7d671-9649-4b43-8325-9b6ceeb732f6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:54.203394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7d671-9649-4b43-8325-9b6ceeb732f6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:54.203699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a7d671-9649-4b43-8325-9b6ceeb732f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.226480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:54.230998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:54.275864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc3829e-b4d7-4753-9aba-22259533cd8b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:54.287216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-61a7d671-9649-4b43-8325-9b6ceeb732f6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:54.294159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-61a7d671-9649-4b43-8325-9b6ceeb732f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:54.309266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:54.309929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:54.313407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 982/1968] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:54.318305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7d671-9649-4b43-8325-9b6ceeb732f6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:54.323747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:54.324895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] PATH_INFO: `/v3/projects/72f058280223479fb7853896b7fd7394` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.327915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=72f058280223479fb7853896b7fd7394)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:54.336594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a7d671-9649-4b43-8325-9b6ceeb732f6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:54.356950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:54.373272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:54.373906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:54.374183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:54.374452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.375110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:54.375411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:54.381559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:54.389570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:54.404049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 983/1969] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:54.408583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:54.426756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b85b74-cd3d-4d8e-a4d8-25e77a8164fc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:54.437623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:54.440299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:54.440589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:54.440866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.441514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:54.450103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72f058280223479fb7853896b7fd7394', 'request_id': 'req-49b281d1-19bc-42e9-84a0-193d2f0575dd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.451675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:54.458916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72f058280223479fb7853896b7fd7394', 'request_id': 'req-49b281d1-19bc-42e9-84a0-193d2f0575dd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.467239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72f058280223479fb7853896b7fd7394', 'request_id': 'req-49b281d1-19bc-42e9-84a0-193d2f0575dd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.489809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] The token cache is being invalidate because project 72f058280223479fb7853896b7fd7394 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:54.490451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.491528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.496632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 987/1970] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:54 2022] DELETE /identity/v3/projects/72f058280223479fb7853896b7fd7394 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:54.497630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:54.499062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:54.503031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 984/1971] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:54 2022] POST /identity/v3/projects => generated 371 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:54.508835 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:54.510608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:54.636323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b281d1-19bc-42e9-84a0-193d2f0575dd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:54.640662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec48ac-377a-4a2e-8008-55c4e16c98e2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:54.641518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:54.642362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:54.642646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:54.643056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] PATH_INFO: `/v3/users/33698dbcf4b04d168e8eda15a30c495f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.646435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] RBAC: Authorizing `identity:delete_user(user_id=33698dbcf4b04d168e8eda15a30c495f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:54.649300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:54.653658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:54.655582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:54.655866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:54.656163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.657833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:54.665735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:54.704553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:54.725412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 985/1972] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:54 2022] POST /identity/v3/users => generated 286 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:54:54.728204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:54.749629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33698dbcf4b04d168e8eda15a30c495f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.750441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33698dbcf4b04d168e8eda15a30c495f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.758444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3acb7d1-e252-4c32-991b-a1c0df55f357 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:54.759041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33698dbcf4b04d168e8eda15a30c495f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.780215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:54.780215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:54.780215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:54.780215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:54.780591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:54.780591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:54.780744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33698dbcf4b04d168e8eda15a30c495f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.791959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33698dbcf4b04d168e8eda15a30c495f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:54.825874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 988/1973] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:54 2022] DELETE /identity/v3/users/33698dbcf4b04d168e8eda15a30c495f => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:54.835233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:54.838769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 986/1974] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:54 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:54.839780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:54.894403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:55.012583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533bed64-f510-4a6a-aaae-d4cca7063a9f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.021483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855f646a-fd96-4b72-ab95-615f0912193a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.022588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855f646a-fd96-4b72-ab95-615f0912193a admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.022871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855f646a-fd96-4b72-ab95-615f0912193a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.023141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855f646a-fd96-4b72-ab95-615f0912193a admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.023979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855f646a-fd96-4b72-ab95-615f0912193a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.026005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855f646a-fd96-4b72-ab95-615f0912193a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.039979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 989/1975] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:54 2022] GET /identity/v3/roles => generated 1420 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:55.045096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.066531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e21de2-4024-4f66-9eb3-f2181eb7881d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.071988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.072656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.072934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.074807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] PATH_INFO: `/v3/projects/3f342e4529b64add8e216082ef9dff14/users/dbd7d1a148fd47d9b6f88d8c988182bd/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.099344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3f342e4529b64add8e216082ef9dff14, user_id=dbd7d1a148fd47d9b6f88d8c988182bd, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.105488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.127413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 987/1976] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:55 2022] PUT /identity/v3/projects/3f342e4529b64add8e216082ef9dff14/users/dbd7d1a148fd47d9b6f88d8c988182bd/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:55.133671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.133960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.134236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.156453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:55.158056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:55.228286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 990/1977] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:55.239796 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.241545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:55.279326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.282119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:55.299691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71cca25-5aca-4b8b-a196-ce59b68092f6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.304968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.305597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.305875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.306145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] PATH_INFO: `/v3/projects/500b5a975ae64af08e4443cd284ad02a` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.307249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] RBAC: Authorizing `identity:delete_project(project_id=500b5a975ae64af08e4443cd284ad02a)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.309322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.340571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c67bb40-cc02-4cbd-baf8-38de1bafd55f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.344193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c97a40b-c99f-4325-ae45-116069e66fd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.344798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c97a40b-c99f-4325-ae45-116069e66fd0 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.345062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c97a40b-c99f-4325-ae45-116069e66fd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.345324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c97a40b-c99f-4325-ae45-116069e66fd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.354724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c97a40b-c99f-4325-ae45-116069e66fd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.356762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c97a40b-c99f-4325-ae45-116069e66fd0 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.359600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '500b5a975ae64af08e4443cd284ad02a', 'request_id': 'req-f56980ae-3b6c-407b-b808-62edfa0a4910'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:55.366010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '500b5a975ae64af08e4443cd284ad02a', 'request_id': 'req-f56980ae-3b6c-407b-b808-62edfa0a4910'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:55.367525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 988/1978] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:55.371624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '500b5a975ae64af08e4443cd284ad02a', 'request_id': 'req-f56980ae-3b6c-407b-b808-62edfa0a4910'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:55.384381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] The token cache is being invalidate because project 500b5a975ae64af08e4443cd284ad02a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:55.384670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:55.385312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f56980ae-3b6c-407b-b808-62edfa0a4910 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:55.386953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 991/1979] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:55 2022] DELETE /identity/v3/projects/500b5a975ae64af08e4443cd284ad02a => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:55.393210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2ef32e-d102-44f7-92d8-5d8dab1f2837 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.393519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2ef32e-d102-44f7-92d8-5d8dab1f2837 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.393762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2ef32e-d102-44f7-92d8-5d8dab1f2837 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.413829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ea2ef32e-d102-44f7-92d8-5d8dab1f2837 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:55.415029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ea2ef32e-d102-44f7-92d8-5d8dab1f2837 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:55.484391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 989/1980] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:55.494176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007bd44c-8230-4936-835b-e0faecd3098c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.494457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007bd44c-8230-4936-835b-e0faecd3098c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.494724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007bd44c-8230-4936-835b-e0faecd3098c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.515236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-007bd44c-8230-4936-835b-e0faecd3098c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:55.516616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-007bd44c-8230-4936-835b-e0faecd3098c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:55.525823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 992/1981] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:55.530121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2ef32e-d102-44f7-92d8-5d8dab1f2837 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.541374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2ef32e-d102-44f7-92d8-5d8dab1f2837 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.546958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.547610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.547897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.548273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.549004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:54:55.549221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.550971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.564727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 990/1982] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:54:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:54:55.569524 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007bd44c-8230-4936-835b-e0faecd3098c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.579457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007bd44c-8230-4936-835b-e0faecd3098c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.585064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.585669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.585948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.586216 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.586880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.588962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.631983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 993/1983] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:55 2022] POST /identity/v3/projects => generated 387 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:55.633989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.646248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100bc70a-5231-403b-8ece-019265600668 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.747802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:55.777948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.778644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.779384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.780202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.782520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.784797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.870528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 991/1984] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:55 2022] POST /identity/v3/users => generated 294 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:55.873329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.895983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbf3b8-b07e-48cc-893e-88db6ae6d08a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.906973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.908095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.908599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.909943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.911525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.913642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.932049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 994/1985] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:55 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:55.932482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:55.936051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.951025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0883e83c-54e7-4596-972c-16e8e1ff9d07 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:55.957282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4433b2-620f-4224-82d0-83ed8a873b46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:55.958885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4433b2-620f-4224-82d0-83ed8a873b46 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:55.959432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4433b2-620f-4224-82d0-83ed8a873b46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:55.959893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4433b2-620f-4224-82d0-83ed8a873b46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:55.960660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4433b2-620f-4224-82d0-83ed8a873b46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:55.962497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4433b2-620f-4224-82d0-83ed8a873b46 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:55.980232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 992/1986] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:55 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:55.981692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:55.994948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377924d3-1f0a-4026-a1e3-7f72e72d4518 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:56.001069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:56.001765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.002369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.002489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] PATH_INFO: `/v3/projects/4b7e0d62ee844050bf86a27c8806edf8/users/c9c7dd67e5684b3385996f9dd5f2b356/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.061965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b7e0d62ee844050bf86a27c8806edf8, user_id=c9c7dd67e5684b3385996f9dd5f2b356, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:56.064721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:56.102870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 995/1987] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:55 2022] PUT /identity/v3/projects/4b7e0d62ee844050bf86a27c8806edf8/users/c9c7dd67e5684b3385996f9dd5f2b356/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:56.121051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6234579c-712b-4e3a-86b4-f6a3df3f5787 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.121341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6234579c-712b-4e3a-86b4-f6a3df3f5787 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.121638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6234579c-712b-4e3a-86b4-f6a3df3f5787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.209040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6234579c-712b-4e3a-86b4-f6a3df3f5787 None None] MFA Rules not processed for user `c9c7dd67e5684b3385996f9dd5f2b356`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:56.210492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6234579c-712b-4e3a-86b4-f6a3df3f5787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:56.375529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 993/1988] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:56 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:56.381310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:56.525192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.525477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.525746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.557333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e184512f-fbed-42d4-a5b9-b49360809c14 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:56.568884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:56.571023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.572056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.572773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.573747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:56.576326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:56.591333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] MFA Rules not processed for user `dbd7d1a148fd47d9b6f88d8c988182bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:56.592657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:56.614416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 996/1989] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:56 2022] POST /identity/v3/projects => generated 387 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:54:56.619346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:56.715434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 994/1990] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:56 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:56.716402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:56.717976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:56.729339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e220f0b-454b-40f6-9dc3-77a3ccb4d423 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:56.733669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1e8eee-8793-4e2e-af4d-cc9fbfe53302 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:56.734810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:56.735986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.736290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.736563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.737209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:56.741187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:56.743045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:56.743804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.744218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.744587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.745333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:56.748251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:56.773476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 997/1991] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:56 2022] POST /identity/v3/users => generated 300 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:56.777845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:56.792019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd093c8-5134-4a5e-a497-4f0c9c184f21 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:56.799451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 995/1992] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:54:56 2022] POST /identity/v3/projects => generated 373 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:56.801577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:56.803777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:56.805185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.805620 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.806057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.806883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:56.808865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:56.825275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 998/1993] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:56 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:56.829562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:56.842090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb94f312-a570-404e-9732-6cb849bd1f8f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:56.846638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:56.922178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd8e39e-8c7d-40bd-8815-5dab2a94a0e5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:56.935296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:56.935681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.936001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.936274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.937406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:56.939448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:56.951875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:56.981536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 996/1994] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:54:56 2022] POST /identity/v3/users => generated 293 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:54:56.985422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:56.988445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:56.996460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:56.996460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:56.996746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] PATH_INFO: `/v3/projects/0eeebc6c007747fb94a79c9ef7228cac/users/34a8320572f04cdf93b4b64a99761817/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:56.997994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcf535e-56ae-4f70-94ba-0f2bd9cb22ce admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.005088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.005702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.005979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.006245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.006886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.008990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.015963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0eeebc6c007747fb94a79c9ef7228cac, user_id=34a8320572f04cdf93b4b64a99761817, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.018714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.023368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 997/1995] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:56 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:54:57.024584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:57.036199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e553584-6669-4809-ae8a-a2fd74cbb323 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.037655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 999/1996] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:56 2022] PUT /identity/v3/projects/0eeebc6c007747fb94a79c9ef7228cac/users/34a8320572f04cdf93b4b64a99761817/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:57.041769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:57.108797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.109579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.109970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.110343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] PATH_INFO: `/v3/projects/1fa5587735f64d3d95b425da75a1350d/users/0a1cc0e49fd8471abb2f3a4bd0ef0541/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.116953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7389f4b7-0db4-4705-be04-aac752e4b7d6 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.127189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fa5587735f64d3d95b425da75a1350d, user_id=0a1cc0e49fd8471abb2f3a4bd0ef0541, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.130785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.131446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.132355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.132810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.133087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.134048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.137310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.151923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 998/1997] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:57 2022] PUT /identity/v3/projects/1fa5587735f64d3d95b425da75a1350d/users/0a1cc0e49fd8471abb2f3a4bd0ef0541/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:57.152130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1000/1998] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:57 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:57.155077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:57.156966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:57.170167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c957833-ab6f-4f5f-8f4e-0b0bb76535b2 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.243893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a03e0d3-857c-4f0f-85e1-a3a31b1ba241 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.250453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3f00a6-0d53-43c0-9477-a182ef44da01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.250980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3f00a6-0d53-43c0-9477-a182ef44da01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.251173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.251297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3f00a6-0d53-43c0-9477-a182ef44da01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.251534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3f00a6-0d53-43c0-9477-a182ef44da01 admin admin] PATH_INFO: `/v3/projects/0eeebc6c007747fb94a79c9ef7228cac/users/34a8320572f04cdf93b4b64a99761817/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.251993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.252437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.252780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.253534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3f00a6-0d53-43c0-9477-a182ef44da01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0eeebc6c007747fb94a79c9ef7228cac, user_id=34a8320572f04cdf93b4b64a99761817, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.253652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.256311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.257572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3f00a6-0d53-43c0-9477-a182ef44da01 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.267408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 999/1999] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:54:57 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:54:57.272096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1001/2000] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:57 2022] PUT /identity/v3/projects/0eeebc6c007747fb94a79c9ef7228cac/users/34a8320572f04cdf93b4b64a99761817/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:57.276881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:57.283411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.283693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.284011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.292361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d31da3c-5c36-48ab-a5ec-64039e1f8445 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.322934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:57.326396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] MFA Rules not processed for user `34a8320572f04cdf93b4b64a99761817`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:57.328458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:57.362478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.363164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.363497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.363772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] PATH_INFO: `/v3/projects/1fa5587735f64d3d95b425da75a1350d/users/0a1cc0e49fd8471abb2f3a4bd0ef0541/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.366225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fa5587735f64d3d95b425da75a1350d, user_id=0a1cc0e49fd8471abb2f3a4bd0ef0541, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.368918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.379953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1000/2001] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:54:57 2022] PUT /identity/v3/projects/1fa5587735f64d3d95b425da75a1350d/users/0a1cc0e49fd8471abb2f3a4bd0ef0541/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:57.387902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:57.423680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1002/2002] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:57 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:54:57.436460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:57.438406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:57.591281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:57.593197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:57.594264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d820754f-71dd-4653-984d-f56731c2e3c6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.608495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f97733-b7fe-4a4b-a8db-655dafe92e04 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.609226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f97733-b7fe-4a4b-a8db-655dafe92e04 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.609572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f97733-b7fe-4a4b-a8db-655dafe92e04 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.609761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43f5d35-56cd-4e8e-a63c-454e1bc9e16c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:57.610551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f97733-b7fe-4a4b-a8db-655dafe92e04 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.717063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:57.717759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.718137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.718523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] PATH_INFO: `/v3/users/2f7707a279104f248f1c16db2f6cfde7` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.720710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f7707a279104f248f1c16db2f6cfde7)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.729436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f97733-b7fe-4a4b-a8db-655dafe92e04 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:57.730804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.731394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f97733-b7fe-4a4b-a8db-655dafe92e04 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:57.761652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1001/2003] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:57 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:57.820233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f7707a279104f248f1c16db2f6cfde7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:57.821384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f7707a279104f248f1c16db2f6cfde7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:57.823878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f7707a279104f248f1c16db2f6cfde7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:57.837181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f7707a279104f248f1c16db2f6cfde7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:57.853920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f7707a279104f248f1c16db2f6cfde7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:57.886445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1003/2004] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:57 2022] DELETE /identity/v3/users/2f7707a279104f248f1c16db2f6cfde7 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:57.889806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6526901-38c0-4b72-9241-eb7dc0fd32a5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:57.890241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6526901-38c0-4b72-9241-eb7dc0fd32a5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:57.890716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6526901-38c0-4b72-9241-eb7dc0fd32a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:57.913255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b6526901-38c0-4b72-9241-eb7dc0fd32a5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:57.914625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b6526901-38c0-4b72-9241-eb7dc0fd32a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.011565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1002/2005] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:58.023627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:58.025182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.130456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c22a48-5538-4187-95a1-d469f9fd1f8b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:58.137374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:58.137995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:58.138278 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:58.138557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:58.152000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:58.154235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:58.168503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:58.174180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1004/2006] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:58.481731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:58.482061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:58.482343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:58.519664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:58.523455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] MFA Rules not processed for user `0a1cc0e49fd8471abb2f3a4bd0ef0541`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:58.526252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.606110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:58.608428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.625979 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a74241-8479-4bab-9a9b-f85fa2615ebe service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:58.645004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:58.647337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:58.647598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:58.647871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] PATH_INFO: `/v3/users/c9c7dd67e5684b3385996f9dd5f2b356` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:58.649074 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9c7dd67e5684b3385996f9dd5f2b356)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:58.651014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:58.658487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1003/2007] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:58 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:58.664067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:58.665944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.679664 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e550c4d-5291-4107-810c-66c96f557aaf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:58.684544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:58.685548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:58.685834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:58.686097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] PATH_INFO: `/v3/projects/91c9fd811c6848cca297e03cf0da9642` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:58.687325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] RBAC: Authorizing `identity:delete_project(project_id=91c9fd811c6848cca297e03cf0da9642)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:58.689210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:58.719309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9c7dd67e5684b3385996f9dd5f2b356'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.720169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9c7dd67e5684b3385996f9dd5f2b356'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.720865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9c7dd67e5684b3385996f9dd5f2b356'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.729242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9c7dd67e5684b3385996f9dd5f2b356'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.744473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9c7dd67e5684b3385996f9dd5f2b356'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.746125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91c9fd811c6848cca297e03cf0da9642', 'request_id': 'req-9646d8d5-dc9b-47d7-ad8a-931022039c0d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.758728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91c9fd811c6848cca297e03cf0da9642', 'request_id': 'req-9646d8d5-dc9b-47d7-ad8a-931022039c0d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.764450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1005/2008] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:58 2022] DELETE /identity/v3/users/c9c7dd67e5684b3385996f9dd5f2b356 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:58.765936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91c9fd811c6848cca297e03cf0da9642', 'request_id': 'req-9646d8d5-dc9b-47d7-ad8a-931022039c0d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.767009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:58.770228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.780337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] The token cache is being invalidate because project 91c9fd811c6848cca297e03cf0da9642 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:58.780653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.781318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9646d8d5-dc9b-47d7-ad8a-931022039c0d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.782830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1004/2009] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:58 2022] DELETE /identity/v3/projects/91c9fd811c6848cca297e03cf0da9642 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:58.785297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:58.785959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:58.786872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:58.808776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:54:58.810808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.831767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d41bf-8f12-419f-9691-36b5391da36d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:58.836210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:58.836811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:58.837090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:58.837360 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:58.839061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.874648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1005/2010] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:54:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:54:58.875495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:58.876798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.889394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f43934-dbb9-49dd-aaab-f7ed6cb067d3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:58.894039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:58.894677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:58.894956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:58.895788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] PATH_INFO: `/v3/users/19105be8be244c1791367ebb616f38f9` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:58.897186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] RBAC: Authorizing `identity:delete_user(user_id=19105be8be244c1791367ebb616f38f9)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:58.899110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:58.902426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:58.903863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:58.913459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1006/2011] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:58 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:58.914223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:58.916213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:58.916329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:58.954266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:58.955930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19105be8be244c1791367ebb616f38f9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.956987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19105be8be244c1791367ebb616f38f9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.985898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19105be8be244c1791367ebb616f38f9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:58.987185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19105be8be244c1791367ebb616f38f9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.009013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0850e24-81dd-44f0-ba3f-951ea7d7274f service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:59.013181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19105be8be244c1791367ebb616f38f9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.014938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:59.016558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:59.016558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:59.016558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:59.018449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:59.042392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1006/2012] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:58 2022] DELETE /identity/v3/users/19105be8be244c1791367ebb616f38f9 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:59.045744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:59.047582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:59.063189 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1007/2013] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:54:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:54:59.525418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:59.527041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:59.554395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:59.555802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:59.575529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d4e0fb-2de1-471b-a90e-fb303115c6cb service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:59.620877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74887817-e894-4cfb-8f0e-d5d74d671c54 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:59.625746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:59.626494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:59.626894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:59.627337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] PATH_INFO: `/v3/projects/4b7e0d62ee844050bf86a27c8806edf8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:59.628512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b7e0d62ee844050bf86a27c8806edf8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:59.630886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:59.656496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:59.657539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:59.657539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:59.657734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] PATH_INFO: `/v3/projects/64315132b3ab4ec28f514445ba2270e7` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:59.659182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64315132b3ab4ec28f514445ba2270e7)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:59.662740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:59.695750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b7e0d62ee844050bf86a27c8806edf8', 'request_id': 'req-5b217fdc-1e3b-4106-a688-a81048a0b13a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.703060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b7e0d62ee844050bf86a27c8806edf8', 'request_id': 'req-5b217fdc-1e3b-4106-a688-a81048a0b13a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.712609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b7e0d62ee844050bf86a27c8806edf8', 'request_id': 'req-5b217fdc-1e3b-4106-a688-a81048a0b13a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.727592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] The token cache is being invalidate because project 4b7e0d62ee844050bf86a27c8806edf8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:59.727918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.728124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64315132b3ab4ec28f514445ba2270e7', 'request_id': 'req-a1a6ee30-6009-42e7-98b5-e23cdec0a015'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.728604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.730790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1007/2014] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:59 2022] DELETE /identity/v3/projects/4b7e0d62ee844050bf86a27c8806edf8 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:59.734266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:59.735028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64315132b3ab4ec28f514445ba2270e7', 'request_id': 'req-a1a6ee30-6009-42e7-98b5-e23cdec0a015'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.736041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:59.742732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64315132b3ab4ec28f514445ba2270e7', 'request_id': 'req-a1a6ee30-6009-42e7-98b5-e23cdec0a015'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.757708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] The token cache is being invalidate because project 64315132b3ab4ec28f514445ba2270e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:54:59.758225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.759282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.763874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1008/2015] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:54:59 2022] DELETE /identity/v3/projects/64315132b3ab4ec28f514445ba2270e7 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:54:59.764876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:54:59.766425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:54:59.770281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:59.816171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b217fdc-1e3b-4106-a688-a81048a0b13a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:59.843330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a6ee30-6009-42e7-98b5-e23cdec0a015 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:54:59.848467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:59.849097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:59.849377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:59.849647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] PATH_INFO: `/v3/users/c864836a410d411bb929e6a311bc5558` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:59.850787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c864836a410d411bb929e6a311bc5558)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:59.852726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:59.854234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:54:59.875043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:54:59.875692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:54:59.875997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:54:59.876388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] PATH_INFO: `/v3/users/34a8320572f04cdf93b4b64a99761817` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:54:59.877529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] RBAC: Authorizing `identity:delete_user(user_id=34a8320572f04cdf93b4b64a99761817)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:54:59.880413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:54:59.904642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c864836a410d411bb929e6a311bc5558'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.905390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c864836a410d411bb929e6a311bc5558'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.906099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c864836a410d411bb929e6a311bc5558'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.913855 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c864836a410d411bb929e6a311bc5558'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.928459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c864836a410d411bb929e6a311bc5558'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.930188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34a8320572f04cdf93b4b64a99761817'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.930874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34a8320572f04cdf93b4b64a99761817'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.938695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34a8320572f04cdf93b4b64a99761817'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.939444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34a8320572f04cdf93b4b64a99761817'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.947427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1009/2016] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:59 2022] DELETE /identity/v3/users/c864836a410d411bb929e6a311bc5558 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:54:59.952319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34a8320572f04cdf93b4b64a99761817'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:54:59.966503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1008/2017] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:54:59 2022] DELETE /identity/v3/users/34a8320572f04cdf93b4b64a99761817 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:00.272161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:00.273618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:00.376593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776aa1d2-42d9-4ce2-9175-02bbdc6320b1 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:00.455233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:00.455867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.456183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.456459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] PATH_INFO: `/v3/projects/e95a767470864a4296f963b2769727b5` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.458174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e95a767470864a4296f963b2769727b5)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:00.460289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:00.494820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:00.518639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e95a767470864a4296f963b2769727b5', 'request_id': 'req-731bf85a-f5ff-4b78-9243-8f98a2e34538'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.526623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e95a767470864a4296f963b2769727b5', 'request_id': 'req-731bf85a-f5ff-4b78-9243-8f98a2e34538'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.531037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:00.532857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:00.536236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e95a767470864a4296f963b2769727b5', 'request_id': 'req-731bf85a-f5ff-4b78-9243-8f98a2e34538'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.549350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c77ade1-feeb-4619-a1cc-2a7464d9778f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:00.560264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] The token cache is being invalidate because project e95a767470864a4296f963b2769727b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:00.561004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.561367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-731bf85a-f5ff-4b78-9243-8f98a2e34538 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.568422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1010/2018] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:00 2022] DELETE /identity/v3/projects/e95a767470864a4296f963b2769727b5 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:00.570002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb7459-1861-4179-946b-6216b8623733 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.570424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb7459-1861-4179-946b-6216b8623733 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.570758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cb7459-1861-4179-946b-6216b8623733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.601481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-30cb7459-1861-4179-946b-6216b8623733 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:00.602779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-30cb7459-1861-4179-946b-6216b8623733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:00.613375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:00.625441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:00.626076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.626355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.626630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] PATH_INFO: `/v3/projects/0eeebc6c007747fb94a79c9ef7228cac` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.627738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0eeebc6c007747fb94a79c9ef7228cac)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:00.630555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:00.680476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1011/2019] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:00.682245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0eeebc6c007747fb94a79c9ef7228cac', 'request_id': 'req-22c09359-75b4-47ed-bd1b-a53b10c9db73'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.688247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a54094-6f48-4c9f-a0fa-863d5458d256 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.688536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a54094-6f48-4c9f-a0fa-863d5458d256 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.688849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a54094-6f48-4c9f-a0fa-863d5458d256 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.689700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0eeebc6c007747fb94a79c9ef7228cac', 'request_id': 'req-22c09359-75b4-47ed-bd1b-a53b10c9db73'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.695981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0eeebc6c007747fb94a79c9ef7228cac', 'request_id': 'req-22c09359-75b4-47ed-bd1b-a53b10c9db73'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.709532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-87a54094-6f48-4c9f-a0fa-863d5458d256 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:00.709706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] The token cache is being invalidate because project 0eeebc6c007747fb94a79c9ef7228cac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:00.709706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.710133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-87a54094-6f48-4c9f-a0fa-863d5458d256 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:00.710308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22c09359-75b4-47ed-bd1b-a53b10c9db73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:00.712216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1009/2020] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:00 2022] DELETE /identity/v3/projects/0eeebc6c007747fb94a79c9ef7228cac => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:00.716475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d2935-7572-4b7a-87ce-fc73dfd36a39 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.716828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d2935-7572-4b7a-87ce-fc73dfd36a39 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.717192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14d2935-7572-4b7a-87ce-fc73dfd36a39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.739197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b14d2935-7572-4b7a-87ce-fc73dfd36a39 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:00.740718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b14d2935-7572-4b7a-87ce-fc73dfd36a39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:00.786592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1012/2021] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:00.792033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a54094-6f48-4c9f-a0fa-863d5458d256 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:00.801887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a54094-6f48-4c9f-a0fa-863d5458d256 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:00.806972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:00.818434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.818729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.819043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.819727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:00.820015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:00.821630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:00.828151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1010/2022] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:00.836336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e86f1-8b7f-4cfe-b00a-ce427efa681e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.836764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e86f1-8b7f-4cfe-b00a-ce427efa681e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.837120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e86f1-8b7f-4cfe-b00a-ce427efa681e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.850562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1013/2023] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:00.854980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:00.861058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-735e86f1-8b7f-4cfe-b00a-ce427efa681e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:00.862328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-735e86f1-8b7f-4cfe-b00a-ce427efa681e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:00.867282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f080f2-3043-4f23-a034-a321ec04afcd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:00.870647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1011/2024] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:00.872459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:00.873991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.874448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.874868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.874992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e86f1-8b7f-4cfe-b00a-ce427efa681e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:00.876008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:00.878067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:00.884785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e86f1-8b7f-4cfe-b00a-ce427efa681e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:00.889505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:00.890192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.890538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.890872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.891572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:00.891997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:00.893652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:00.907036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1012/2025] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:00.911399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:00.916505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1014/2026] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:00 2022] POST /identity/v3/projects => generated 373 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:00.918904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:00.921296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8ef86a-2852-4eda-9e63-fd6baaf63456 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:00.949497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:00.990660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:00.991300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.991581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.991692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1b7999-5862-49c5-8598-abcd59b29196 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:00.991854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:00.992515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:00.994478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:00.997934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:00.998581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:00.998860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:00.999299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.000316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.001043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:01.002465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.024356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1013/2027] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:00 2022] POST /identity/v3/projects => generated 385 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:01.029279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1015/2028] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:00 2022] POST /identity/v3/users => generated 287 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:01.029279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.030353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.039154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e35e80-7ac0-4d64-af30-c75bb211592f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.095063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cda7499-4595-4f3b-b440-448613d5ed94 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.098448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.099056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.099453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.099649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.100311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.100454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.100908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.101184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.101455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.102085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.102419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.103698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.113635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1014/2029] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:01 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:01.118234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.128555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1016/2030] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:01 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:01.132322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700f30a1-12f6-4d7a-837c-318de59a8ce3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.133032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.137282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.137877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.138183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.138456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.139090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.140719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.142609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f52e95-cd3c-4182-9c39-6a0670bd10ef admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.147778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.148539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.148696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.148971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.149628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.151262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.152260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1015/2031] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:01 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:01.156091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.164674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1017/2032] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:01.165760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.169000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb43596-8305-4362-970b-74f0cee4f907 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.173631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb6b824-e8d6-402f-b807-a509377699d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.174268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb6b824-e8d6-402f-b807-a509377699d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.174546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb6b824-e8d6-402f-b807-a509377699d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.174816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb6b824-e8d6-402f-b807-a509377699d3 admin admin] PATH_INFO: `/v3/projects/aad64953b4b44fa780d7f450601df556/users/8f620c5ede5645e6a67d1b5246402947/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.175378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f987e8-2a33-4aeb-a58d-69beb5b88335 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.180495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.181078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.181363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.181638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.182323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.183891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.189930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb6b824-e8d6-402f-b807-a509377699d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aad64953b4b44fa780d7f450601df556, user_id=8f620c5ede5645e6a67d1b5246402947, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.193353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb6b824-e8d6-402f-b807-a509377699d3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.195603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1018/2033] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:01.199665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.209926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1016/2034] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:01 2022] PUT /identity/v3/projects/aad64953b4b44fa780d7f450601df556/users/8f620c5ede5645e6a67d1b5246402947/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:01.211254 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a98a8f-a851-4788-8ca7-097f41bc4d51 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.220357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.220691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.221013 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.249784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] MFA Rules not processed for user `8f620c5ede5645e6a67d1b5246402947`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:01.251676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:01.274961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99659e01-a518-4276-82bc-f7941ebe6a78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.275537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99659e01-a518-4276-82bc-f7941ebe6a78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.275818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99659e01-a518-4276-82bc-f7941ebe6a78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.276159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99659e01-a518-4276-82bc-f7941ebe6a78 admin admin] PATH_INFO: `/v3/projects/1d44219808dc40dbaedd00b9b003b79b/users/e1e05cc6adda4740bfee94dd0fe5adb0/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.291086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99659e01-a518-4276-82bc-f7941ebe6a78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d44219808dc40dbaedd00b9b003b79b, user_id=e1e05cc6adda4740bfee94dd0fe5adb0, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.293859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99659e01-a518-4276-82bc-f7941ebe6a78 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.307103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1019/2035] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:01 2022] PUT /identity/v3/projects/1d44219808dc40dbaedd00b9b003b79b/users/e1e05cc6adda4740bfee94dd0fe5adb0/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:01.317570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e02b8-94da-4eed-beb2-7a3bc8fe6b20 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.317851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e02b8-94da-4eed-beb2-7a3bc8fe6b20 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.318118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e02b8-94da-4eed-beb2-7a3bc8fe6b20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.324995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:01.333480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1017/2036] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:01 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:01.345214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.346795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:01.360207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-fc3e02b8-94da-4eed-beb2-7a3bc8fe6b20 None None] MFA Rules not processed for user `e1e05cc6adda4740bfee94dd0fe5adb0`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:01.361713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fc3e02b8-94da-4eed-beb2-7a3bc8fe6b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:01.400925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:01.418001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84adc5de-5bf0-4a81-b6f6-31851f6ae01f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.445244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1020/2037] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:01 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:01.450727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3e02b8-94da-4eed-beb2-7a3bc8fe6b20 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.499586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.500352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.500737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.504068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.588461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3e02b8-94da-4eed-beb2-7a3bc8fe6b20 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.595443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.596373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.596756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.597126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.597870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.600797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.657276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.659014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.659288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1021/2038] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:01 2022] POST /identity/v3/projects => generated 381 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:01.660370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.680296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1018/2039] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:01 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:01.721476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.741897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ea4b34-149f-4583-8966-019e8d35bf98 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.787436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa66a3-5e08-45bb-8b9a-e355d0a5a475 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.794128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.794744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.795010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.797744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.797744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.800661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.827403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1022/2040] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:01 2022] POST /identity/v3/users => generated 297 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:01.831496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.844187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bccbc8-9ca9-4baf-8348-de0db6cb83f5 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.866425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.867676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.868523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.868897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.869824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.870201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.872864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.873143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.873547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] PATH_INFO: `/v3/users/8f620c5ede5645e6a67d1b5246402947` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.876769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.877226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f620c5ede5645e6a67d1b5246402947)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.887126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:01.887434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:01.914499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1023/2041] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:01 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:01.918987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:01.940110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8af496b-1b56-4201-8b87-fe590d8c2aa4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:01.948062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:01.949384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:01.949755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:01.950149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] PATH_INFO: `/v3/projects/9de663af1e6741529066cd6ef30382f7/users/fe15214ef55e4f06ad57eb488c81ebbd/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:01.986049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f620c5ede5645e6a67d1b5246402947'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:01.986712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f620c5ede5645e6a67d1b5246402947'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:01.988982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9de663af1e6741529066cd6ef30382f7, user_id=fe15214ef55e4f06ad57eb488c81ebbd, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:01.991021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:02.002720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:02.003106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f620c5ede5645e6a67d1b5246402947'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:02.003927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f620c5ede5645e6a67d1b5246402947'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:02.025165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-165aeb5c-0be9-48b1-91a4-86df524d2fc4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f620c5ede5645e6a67d1b5246402947'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:02.032313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1024/2042] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:01 2022] PUT /identity/v3/projects/9de663af1e6741529066cd6ef30382f7/users/fe15214ef55e4f06ad57eb488c81ebbd/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:02.036774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:02.039209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.046782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1019/2043] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:01 2022] DELETE /identity/v3/users/8f620c5ede5645e6a67d1b5246402947 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:02.056241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:02.056727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:02.057073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:02.088053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:02.089639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.166262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058efb19-4baa-404e-b61b-cdaac5b0e423 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:02.237921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1020/2044] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:02.260374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:02.262106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.265234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:02.265916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:02.266203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:02.267771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:02.267771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:02.268991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:02.286561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1025/2045] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:02 2022] GET /identity/v3/roles => generated 1420 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:02.290528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:02.303003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091416bd-1697-4a09-91f5-2f388f079d22 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:02.309344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e28727-4345-4e53-a637-2c651c40f96d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:02.309832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e28727-4345-4e53-a637-2c651c40f96d admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:02.310212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e28727-4345-4e53-a637-2c651c40f96d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:02.310559 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e28727-4345-4e53-a637-2c651c40f96d admin admin] PATH_INFO: `/v3/projects/9de663af1e6741529066cd6ef30382f7/users/fe15214ef55e4f06ad57eb488c81ebbd/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:02.315898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e28727-4345-4e53-a637-2c651c40f96d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9de663af1e6741529066cd6ef30382f7, user_id=fe15214ef55e4f06ad57eb488c81ebbd, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:02.329535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e28727-4345-4e53-a637-2c651c40f96d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:02.342913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31063058-7083-4bc8-b049-5dd79796f845 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:02.353272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1026/2046] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:02 2022] PUT /identity/v3/projects/9de663af1e6741529066cd6ef30382f7/users/fe15214ef55e4f06ad57eb488c81ebbd/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:02.372301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:02.372591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:02.372867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:02.422732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] MFA Rules not processed for user `fe15214ef55e4f06ad57eb488c81ebbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:02.424400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.445878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:02.446661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:02.447001 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:02.447206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:02.512298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:02.522289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:02.559801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1027/2047] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:02 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:02.573169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:02.575547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.577717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:02.579508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:02.591721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1021/2048] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:55:02.610596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:02.613059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.663557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291c05b4-f47e-49ae-88cc-70e1f2518f04 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:02.668523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:02.669272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:02.669614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:02.669982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:02.672034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.705667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba05c643-2605-49bc-82de-7ac538cc7ec8 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:02.710684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:02.711433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:02.711726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:02.712028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:02.714518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:02.749191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:02.750632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:02.762573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1028/2049] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:02 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:02.818247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:02.819838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:02.839659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1022/2050] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:02 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:03.243950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:03.245850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:03.282726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723f010c-c12a-403d-9ebf-8a90c71ddc41 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:03.288142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:03.288767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:03.289069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:03.289333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] PATH_INFO: `/v3/projects/aad64953b4b44fa780d7f450601df556` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:03.290513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aad64953b4b44fa780d7f450601df556)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:03.292854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:03.345048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aad64953b4b44fa780d7f450601df556', 'request_id': 'req-1085256a-b5e0-4edb-b401-2f4724a613d9'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:03.352499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aad64953b4b44fa780d7f450601df556', 'request_id': 'req-1085256a-b5e0-4edb-b401-2f4724a613d9'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:03.358558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aad64953b4b44fa780d7f450601df556', 'request_id': 'req-1085256a-b5e0-4edb-b401-2f4724a613d9'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:03.372358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] The token cache is being invalidate because project aad64953b4b44fa780d7f450601df556 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:03.372665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:03.373375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1085256a-b5e0-4edb-b401-2f4724a613d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:03.374895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1029/2051] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:03 2022] DELETE /identity/v3/projects/aad64953b4b44fa780d7f450601df556 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:05.265229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:05.266675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:05.281280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444a1ef2-6582-4b2b-ae59-50f1fca7f4c4 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:05.347742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:05.348546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:05.348824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:05.349094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] PATH_INFO: `/v3/users/d407a4aee9b349518e71104f03549900` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:05.350239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d407a4aee9b349518e71104f03549900)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:05.352196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:05.385406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:05.410226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd407a4aee9b349518e71104f03549900'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:05.411793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd407a4aee9b349518e71104f03549900'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:05.421657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd407a4aee9b349518e71104f03549900'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:05.422547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd407a4aee9b349518e71104f03549900'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:05.443821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd407a4aee9b349518e71104f03549900'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:05.468056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1023/2052] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:05 2022] DELETE /identity/v3/users/d407a4aee9b349518e71104f03549900 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:05.473561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:05.474384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:05.474811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:05.500340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:05.502026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:05.529635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:05.630940 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1030/2053] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:05.639665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:05.643395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:05.711153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc807553-2b95-42e4-b666-1ac84a8d8012 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:05.717096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:05.717900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:05.718552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:05.718833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:05.729630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:05.731150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:05.750557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1024/2054] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:55:06.073655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:06.075562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:06.094503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b03105-6b30-4cef-a92d-b9d7c78c662c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:06.100615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:06.101597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:06.102084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:06.102560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] PATH_INFO: `/v3/projects/2006fddfe87b49378370b389b68a3695` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:06.104331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2006fddfe87b49378370b389b68a3695)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:06.106723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:06.182256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2006fddfe87b49378370b389b68a3695', 'request_id': 'req-4ce7c7b4-52c9-4288-b019-696db4aa2854'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.193488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2006fddfe87b49378370b389b68a3695', 'request_id': 'req-4ce7c7b4-52c9-4288-b019-696db4aa2854'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.202243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2006fddfe87b49378370b389b68a3695', 'request_id': 'req-4ce7c7b4-52c9-4288-b019-696db4aa2854'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.222925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] The token cache is being invalidate because project 2006fddfe87b49378370b389b68a3695 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:06.223246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.223873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.225831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1031/2055] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:06 2022] DELETE /identity/v3/projects/2006fddfe87b49378370b389b68a3695 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:06.230432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:06.250594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c67b46-7335-4cb8-9c90-a559a11c5345 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:06.389922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:06.390637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:06.390926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:06.391290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] PATH_INFO: `/v3/users/2d1e93cb9f4742ae8cfb40370de44b22` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:06.392511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d1e93cb9f4742ae8cfb40370de44b22)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:06.396506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:06.476655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d1e93cb9f4742ae8cfb40370de44b22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.480534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d1e93cb9f4742ae8cfb40370de44b22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.483037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:06.500024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d1e93cb9f4742ae8cfb40370de44b22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.501436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d1e93cb9f4742ae8cfb40370de44b22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.520434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61c14501-1c28-4ec4-98f6-4042c819dc10 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d1e93cb9f4742ae8cfb40370de44b22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:06.540345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1025/2056] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:06 2022] DELETE /identity/v3/users/2d1e93cb9f4742ae8cfb40370de44b22 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:07.156727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.159071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:07.179025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce7c7b4-52c9-4288-b019-696db4aa2854 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.290750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.291399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.291703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.291992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] PATH_INFO: `/v3/projects/e38dd123414b40c787869d00de4fb87b` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.293595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e38dd123414b40c787869d00de4fb87b)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.298600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.381103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e38dd123414b40c787869d00de4fb87b', 'request_id': 'req-894fa30d-efaf-4123-b232-20388ed60e5c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:07.382527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:07.393647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e38dd123414b40c787869d00de4fb87b', 'request_id': 'req-894fa30d-efaf-4123-b232-20388ed60e5c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:07.406874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e38dd123414b40c787869d00de4fb87b', 'request_id': 'req-894fa30d-efaf-4123-b232-20388ed60e5c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:07.440195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] The token cache is being invalidate because project e38dd123414b40c787869d00de4fb87b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:07.440195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:07.440195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:07.440195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1032/2057] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:07 2022] DELETE /identity/v3/projects/e38dd123414b40c787869d00de4fb87b => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:07.440476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a377ccd-b89e-49de-bcc6-00e09f1146d4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.440476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a377ccd-b89e-49de-bcc6-00e09f1146d4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.440476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a377ccd-b89e-49de-bcc6-00e09f1146d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.464884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9a377ccd-b89e-49de-bcc6-00e09f1146d4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:07.466234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9a377ccd-b89e-49de-bcc6-00e09f1146d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:07.522787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.524712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:07.546559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1026/2058] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:07.556618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad31ec0a-f7d7-42b2-b9f6-19ae3804d3e6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.557134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad31ec0a-f7d7-42b2-b9f6-19ae3804d3e6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.557409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad31ec0a-f7d7-42b2-b9f6-19ae3804d3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.578483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ad31ec0a-f7d7-42b2-b9f6-19ae3804d3e6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:07.579695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ad31ec0a-f7d7-42b2-b9f6-19ae3804d3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:07.588407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1027/2059] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:07.593133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad31ec0a-f7d7-42b2-b9f6-19ae3804d3e6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.603511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-894fa30d-efaf-4123-b232-20388ed60e5c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.609651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad31ec0a-f7d7-42b2-b9f6-19ae3804d3e6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.610106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.610955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.611304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.611640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.613469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:07.614707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.615395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.615746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.616171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.616885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:07.617246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.618911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.633298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1028/2060] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:07.638189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.652088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de5b9e0-edb1-4078-9725-c646d86dc126 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.658385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.659086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.659438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.659781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.662204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.663643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.698355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.701242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1029/2061] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:07 2022] POST /identity/v3/projects => generated 389 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:07.702109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.702544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.720064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1033/2062] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:07 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:55:07.791821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50456fc3-74b8-41de-8bcb-62bee149a041 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.801542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.802785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.803071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.803346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.804589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.806684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.835976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1030/2063] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:07 2022] POST /identity/v3/users => generated 295 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:07.839817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.852317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5bbe8a-1885-47fc-b2d5-8bf5d109c7ab service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.857025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.857711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.857995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.858266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.858917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.860706 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.873754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1034/2064] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:07.878032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.890919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04951551-1fcf-4bb1-8ff5-a06ed967623c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.903309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47858745-ff04-44ea-8160-e9c70ec4d5cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.904127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47858745-ff04-44ea-8160-e9c70ec4d5cd admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.904406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47858745-ff04-44ea-8160-e9c70ec4d5cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.904710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47858745-ff04-44ea-8160-e9c70ec4d5cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.905365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47858745-ff04-44ea-8160-e9c70ec4d5cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.906962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47858745-ff04-44ea-8160-e9c70ec4d5cd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.920963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1031/2065] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:07 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:07.925200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:07.936871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cac9ab1-cd6c-4f47-9b0f-454f8eddf63f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:07.941831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:07.942724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.943024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.943316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] PATH_INFO: `/v3/projects/6420947b2dd941ed9b42b4ff13b4dc11/users/a0e129bfbd5947fd9e2dcb694db6f691/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:07.961856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6420947b2dd941ed9b42b4ff13b4dc11, user_id=a0e129bfbd5947fd9e2dcb694db6f691, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:07.964550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:07.983293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1035/2066] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:07 2022] PUT /identity/v3/projects/6420947b2dd941ed9b42b4ff13b4dc11/users/a0e129bfbd5947fd9e2dcb694db6f691/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:07.995779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ba4e0-4fb7-480d-abdb-ec47a27642c4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:07.996124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ba4e0-4fb7-480d-abdb-ec47a27642c4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:07.996425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9ba4e0-4fb7-480d-abdb-ec47a27642c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.005529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=3,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:08.055190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-cd9ba4e0-4fb7-480d-abdb-ec47a27642c4 None None] MFA Rules not processed for user `a0e129bfbd5947fd9e2dcb694db6f691`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:08.056583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cd9ba4e0-4fb7-480d-abdb-ec47a27642c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:08.173034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1032/2067] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:07 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:08.177994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:08.240102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:DELETE=3,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:08.250623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be682a8-b23b-4e16-82d9-03c6dd4b478c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:08.257978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:08.258725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.259013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.259287 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.259973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:08.261914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:08.291310 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1036/2068] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:08 2022] POST /identity/v3/projects => generated 389 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:08.295363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ba4e0-4fb7-480d-abdb-ec47a27642c4 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:08.307370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9ba4e0-4fb7-480d-abdb-ec47a27642c4 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:08.402960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:08.403586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.403866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.404162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.405473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:08.408181 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:08.435260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1033/2069] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:08 2022] POST /identity/v3/users => generated 301 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:08.439637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:08.452611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ed258a-a265-4770-8668-e8c1230b0ad6 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:08.458363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:08.459147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.459555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.459835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.460697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:08.462291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:08.476264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1037/2070] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:08 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:08.481894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:08.494688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad2f07b-d069-473e-9a6b-9b8fd9dc0b3d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:08.499617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:08.500303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.500583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.500853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] PATH_INFO: `/v3/projects/f33f4f8f2ea04d9dacad2d3c1d3839b3/users/1df7d1c483ad4341b6d0a8149651d4ef/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.519087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f33f4f8f2ea04d9dacad2d3c1d3839b3, user_id=1df7d1c483ad4341b6d0a8149651d4ef, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:08.522729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:08.538878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1034/2071] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:08 2022] PUT /identity/v3/projects/f33f4f8f2ea04d9dacad2d3c1d3839b3/users/1df7d1c483ad4341b6d0a8149651d4ef/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:08.542635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:08.553445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b0e0a2-a568-4ccb-b788-4af368029891 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:08.622994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4f00e5-ce42-4eea-91e8-7f684efef33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:08.624102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f00e5-ce42-4eea-91e8-7f684efef33d admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.624387 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f00e5-ce42-4eea-91e8-7f684efef33d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.624660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f00e5-ce42-4eea-91e8-7f684efef33d admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.625325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4f00e5-ce42-4eea-91e8-7f684efef33d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:08.626949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4f00e5-ce42-4eea-91e8-7f684efef33d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:08.642316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1038/2072] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:08 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:08.643282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:08.655042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbfca7b-8290-4793-90d2-833672158f28 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:08.660096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:08.661341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.661659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.662063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] PATH_INFO: `/v3/projects/f33f4f8f2ea04d9dacad2d3c1d3839b3/users/1df7d1c483ad4341b6d0a8149651d4ef/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.664138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f33f4f8f2ea04d9dacad2d3c1d3839b3, user_id=1df7d1c483ad4341b6d0a8149651d4ef, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:08.666838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:08.679779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1035/2073] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:08 2022] PUT /identity/v3/projects/f33f4f8f2ea04d9dacad2d3c1d3839b3/users/1df7d1c483ad4341b6d0a8149651d4ef/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:08.691142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce401de-a73c-4ad0-a443-43965a2bdebc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.691142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce401de-a73c-4ad0-a443-43965a2bdebc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.691142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce401de-a73c-4ad0-a443-43965a2bdebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.722159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8ce401de-a73c-4ad0-a443-43965a2bdebc None None] MFA Rules not processed for user `1df7d1c483ad4341b6d0a8149651d4ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:08.723442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8ce401de-a73c-4ad0-a443-43965a2bdebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:08.809434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1039/2074] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:08 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:08.821036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:08.875180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:08.886378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc0d0b0-821b-4b73-bcb3-40262cf04c86 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:08.890669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09961d17-bbd7-4c3d-83f1-bc0f302aecb2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:08.891347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09961d17-bbd7-4c3d-83f1-bc0f302aecb2 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:08.891724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09961d17-bbd7-4c3d-83f1-bc0f302aecb2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:08.892046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09961d17-bbd7-4c3d-83f1-bc0f302aecb2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:08.907038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09961d17-bbd7-4c3d-83f1-bc0f302aecb2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:08.909739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09961d17-bbd7-4c3d-83f1-bc0f302aecb2 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:08.932092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1036/2075] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:08 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:09.073811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce401de-a73c-4ad0-a443-43965a2bdebc None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:09.137283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:09.162552 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce401de-a73c-4ad0-a443-43965a2bdebc None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:09.167819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:09.168503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:09.169023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:09.169309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] PATH_INFO: `/v3/users/a0e129bfbd5947fd9e2dcb694db6f691` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:09.170812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0e129bfbd5947fd9e2dcb694db6f691)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:09.173009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:09.300595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0e129bfbd5947fd9e2dcb694db6f691'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:09.301490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0e129bfbd5947fd9e2dcb694db6f691'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:09.302078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0e129bfbd5947fd9e2dcb694db6f691'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:09.316676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0e129bfbd5947fd9e2dcb694db6f691'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:09.341456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0e129bfbd5947fd9e2dcb694db6f691'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:09.377345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1040/2076] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:09 2022] DELETE /identity/v3/users/a0e129bfbd5947fd9e2dcb694db6f691 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:09.383232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:09.384129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:09.384549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:09.419660 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:09.421346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:09.563297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1037/2077] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:09.573210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:09.575830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:09.670710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487ee006-9da1-4997-a6b1-47126d054b23 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:09.675258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:09.675969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:09.676357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:09.676512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:09.690415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:09.692602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:09.707604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1041/2078] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:10.104291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:10.105806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:10.135357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48e2af51-1072-4d83-8fe2-8c9683685e0a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:10.140268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:10.140920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:10.141197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:10.141461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] PATH_INFO: `/v3/projects/6420947b2dd941ed9b42b4ff13b4dc11` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:10.142610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6420947b2dd941ed9b42b4ff13b4dc11)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:10.144692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:10.200433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6420947b2dd941ed9b42b4ff13b4dc11', 'request_id': 'req-3f04a77c-0776-42ff-91d3-88f3a2f69152'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.207432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6420947b2dd941ed9b42b4ff13b4dc11', 'request_id': 'req-3f04a77c-0776-42ff-91d3-88f3a2f69152'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.214391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6420947b2dd941ed9b42b4ff13b4dc11', 'request_id': 'req-3f04a77c-0776-42ff-91d3-88f3a2f69152'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.230832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] The token cache is being invalidate because project 6420947b2dd941ed9b42b4ff13b4dc11 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:10.231121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.231777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.234411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1038/2079] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:10 2022] DELETE /identity/v3/projects/6420947b2dd941ed9b42b4ff13b4dc11 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:10.239517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:10.249773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b3112b-cc60-439c-aa02-5bec767adba3 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:10.346170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:10.346959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:10.347467 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:10.348894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] PATH_INFO: `/v3/users/1df7d1c483ad4341b6d0a8149651d4ef` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:10.350232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1df7d1c483ad4341b6d0a8149651d4ef)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:10.352342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:10.373686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:10.431315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1df7d1c483ad4341b6d0a8149651d4ef'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.432285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1df7d1c483ad4341b6d0a8149651d4ef'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.433838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1df7d1c483ad4341b6d0a8149651d4ef'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.444602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1df7d1c483ad4341b6d0a8149651d4ef'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.459992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-80d660e2-885b-4ede-a5bd-38550a4f5ec2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1df7d1c483ad4341b6d0a8149651d4ef'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.474065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1042/2080] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:10 2022] DELETE /identity/v3/users/1df7d1c483ad4341b6d0a8149651d4ef => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:10.795928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:10.797938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:10.816429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f04a77c-0776-42ff-91d3-88f3a2f69152 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:10.896273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:10.897092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:10.897684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:10.897786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] PATH_INFO: `/v3/projects/f33f4f8f2ea04d9dacad2d3c1d3839b3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:10.898857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f33f4f8f2ea04d9dacad2d3c1d3839b3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:10.901073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:10.916515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:10.983261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f33f4f8f2ea04d9dacad2d3c1d3839b3', 'request_id': 'req-22be2b4e-c206-4e21-a335-33a26ee51ab0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:10.993532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f33f4f8f2ea04d9dacad2d3c1d3839b3', 'request_id': 'req-22be2b4e-c206-4e21-a335-33a26ee51ab0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:11.009146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f33f4f8f2ea04d9dacad2d3c1d3839b3', 'request_id': 'req-22be2b4e-c206-4e21-a335-33a26ee51ab0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:11.030122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] The token cache is being invalidate because project f33f4f8f2ea04d9dacad2d3c1d3839b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:11.031428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:11.032034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-22be2b4e-c206-4e21-a335-33a26ee51ab0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:11.039503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1039/2081] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:10 2022] DELETE /identity/v3/projects/f33f4f8f2ea04d9dacad2d3c1d3839b3 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:11.041645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be14ab8-20d2-4e97-9468-ae82a4a7327a None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.041831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be14ab8-20d2-4e97-9468-ae82a4a7327a None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.042213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be14ab8-20d2-4e97-9468-ae82a4a7327a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.066975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5be14ab8-20d2-4e97-9468-ae82a4a7327a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:11.068499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5be14ab8-20d2-4e97-9468-ae82a4a7327a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:11.165722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1043/2082] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:11.173540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ca171f-4f6f-4e41-849c-15a544be9eb2 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.173824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ca171f-4f6f-4e41-849c-15a544be9eb2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.174092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ca171f-4f6f-4e41-849c-15a544be9eb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.198858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f5ca171f-4f6f-4e41-849c-15a544be9eb2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:11.200500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f5ca171f-4f6f-4e41-849c-15a544be9eb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:11.211263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1040/2083] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:11.215955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be14ab8-20d2-4e97-9468-ae82a4a7327a None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.226197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be14ab8-20d2-4e97-9468-ae82a4a7327a None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.231761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.232507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.232792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.233086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.233758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:11.234078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.235629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.248571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1044/2084] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:11.253251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ca171f-4f6f-4e41-849c-15a544be9eb2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.264480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ca171f-4f6f-4e41-849c-15a544be9eb2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.268868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.269623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.270409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.270680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.271428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.273377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.311974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1041/2085] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:11 2022] POST /identity/v3/projects => generated 365 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:11.314521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.326861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bdd0f13-33a1-4e7a-9a80-c9a1ca5e1b9b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.411758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.412401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.412652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.412951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.413571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.415577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.438018 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1045/2086] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:11 2022] POST /identity/v3/users => generated 283 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:11.442213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.453190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020743c0-40b2-4d46-8aa6-f5e617e72b9c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.458536 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.459129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.459405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.459672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.460344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.461976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.472355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1042/2087] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:11.477178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.488028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400ef697-8eb0-40fa-9996-a23572ab7824 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.493003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe2e6a-1eed-4db1-a368-24f2962d0700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.493602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe2e6a-1eed-4db1-a368-24f2962d0700 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.493882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe2e6a-1eed-4db1-a368-24f2962d0700 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.494155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe2e6a-1eed-4db1-a368-24f2962d0700 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.494797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe2e6a-1eed-4db1-a368-24f2962d0700 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.496415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe2e6a-1eed-4db1-a368-24f2962d0700 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.497744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:11.508832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1046/2088] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:11.512799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.524242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba77443-aec3-4c2e-a119-2e697d8dfd5a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.528951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.529588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.529891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.530162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] PATH_INFO: `/v3/projects/ec57ed444f3d4f9a9b4b79df81681160/users/0ee1005a7ae141749869d4578b59c0da/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.546121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec57ed444f3d4f9a9b4b79df81681160, user_id=0ee1005a7ae141749869d4578b59c0da, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.548804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.561726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1043/2089] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:11 2022] PUT /identity/v3/projects/ec57ed444f3d4f9a9b4b79df81681160/users/0ee1005a7ae141749869d4578b59c0da/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:11.573317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e08546-f8df-4703-ac3e-33386723740d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.573534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e08546-f8df-4703-ac3e-33386723740d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.573859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e08546-f8df-4703-ac3e-33386723740d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.602516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c4e08546-f8df-4703-ac3e-33386723740d None None] MFA Rules not processed for user `0ee1005a7ae141749869d4578b59c0da`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:11.603874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c4e08546-f8df-4703-ac3e-33386723740d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:11.668442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1047/2090] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:11 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:11.672978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.734111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45429e14-dc4a-4dab-b44c-f161b68088ad admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.738733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.739315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.739589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.739854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.740508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.742858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.770863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1044/2091] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:11 2022] POST /identity/v3/projects => generated 367 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:11.774930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e08546-f8df-4703-ac3e-33386723740d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.785922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e08546-f8df-4703-ac3e-33386723740d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.843785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.844353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.844635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.844906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.845526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.847435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.868941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1048/2092] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:11 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:11.873646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.886512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad5fd3-e649-463b-a78b-e4eb4cdc3a40 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.891500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.892308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.892764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.893087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.894040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.896951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.908551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1045/2093] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:11 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:11.912854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.922959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc81be2-f83e-4077-bd3e-e204f3263c21 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.927704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:11.928375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:11.928716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:11.928995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] PATH_INFO: `/v3/projects/17214a73de22428eb996f719faad30fc/users/bd9872df9cd940fc959364f22bfe04e8/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:11.946012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] RBAC: Authorizing `identity:create_grant(project_id=17214a73de22428eb996f719faad30fc, user_id=bd9872df9cd940fc959364f22bfe04e8, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:11.948686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:11.961676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1049/2094] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:11 2022] PUT /identity/v3/projects/17214a73de22428eb996f719faad30fc/users/bd9872df9cd940fc959364f22bfe04e8/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:11.965544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:11.975637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7615de0-6143-4429-b035-596b16b7e0a9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:11.983258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:12.045587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dc2b24-028c-4658-ac14-719102f1a2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:12.046193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc2b24-028c-4658-ac14-719102f1a2da admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:12.046464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc2b24-028c-4658-ac14-719102f1a2da admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:12.046729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dc2b24-028c-4658-ac14-719102f1a2da admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:12.047358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dc2b24-028c-4658-ac14-719102f1a2da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:12.049101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05dc2b24-028c-4658-ac14-719102f1a2da admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:12.060019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1046/2095] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:11 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:12.064246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:12.075875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653fe9a8-0029-4f11-8513-afc6b3f2d65d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:12.080893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:12.081567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:12.081839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:12.082103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] PATH_INFO: `/v3/projects/17214a73de22428eb996f719faad30fc/users/bd9872df9cd940fc959364f22bfe04e8/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:12.084687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] RBAC: Authorizing `identity:create_grant(project_id=17214a73de22428eb996f719faad30fc, user_id=bd9872df9cd940fc959364f22bfe04e8, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:12.087400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:12.098517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1050/2096] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:12 2022] PUT /identity/v3/projects/17214a73de22428eb996f719faad30fc/users/bd9872df9cd940fc959364f22bfe04e8/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:12.108340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:12.108624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:12.108919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:12.139164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] MFA Rules not processed for user `bd9872df9cd940fc959364f22bfe04e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:12.140459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:12.214427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1047/2097] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:12 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:12.226803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:12.228884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:12.274132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:12.291759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faca4596-b936-449f-961f-a8c5547c8cac admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:12.295785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:12.296513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:12.296860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:12.297242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:12.308126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:12.309497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:12.318987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1051/2098] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:12 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:55:12.403176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:12.404896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:12.417753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507ee7dd-9d1f-44fb-bca7-fed21b720f4c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:12.421789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6fb314-5a78-4cc9-8b6d-722ee5f22226 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:12.422483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6fb314-5a78-4cc9-8b6d-722ee5f22226 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:12.422882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6fb314-5a78-4cc9-8b6d-722ee5f22226 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:12.423216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6fb314-5a78-4cc9-8b6d-722ee5f22226 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:12.491014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6fb314-5a78-4cc9-8b6d-722ee5f22226 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:12.492510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6fb314-5a78-4cc9-8b6d-722ee5f22226 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:12.501762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1048/2099] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:12 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:13.149985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:13.151451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:13.218581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a59b240-dff2-4cda-9912-c9927439d5f6 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:13.223961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:13.224589 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:13.224865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:13.225132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] PATH_INFO: `/v3/users/e1e05cc6adda4740bfee94dd0fe5adb0` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:13.226228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1e05cc6adda4740bfee94dd0fe5adb0)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:13.228224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:13.274816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1e05cc6adda4740bfee94dd0fe5adb0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.275464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1e05cc6adda4740bfee94dd0fe5adb0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.276094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1e05cc6adda4740bfee94dd0fe5adb0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.283217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1e05cc6adda4740bfee94dd0fe5adb0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.296311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1e05cc6adda4740bfee94dd0fe5adb0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.309756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1052/2100] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:13 2022] DELETE /identity/v3/users/e1e05cc6adda4740bfee94dd0fe5adb0 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:13.314386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:13.314887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:13.314981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:13.335203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:13.336530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:13.364839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:13.402108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1049/2101] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:13.412356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:13.413789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:13.477561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d76fa-9bfa-4ab0-9889-dd03bb1314e3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:13.482301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:13.482858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:13.483126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:13.483618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:13.494861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:13.496325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:13.510640 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1053/2102] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:13.789838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:13.791247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:13.805386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ac8ed8-fdb0-4542-9b09-23614c65477b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:13.814329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:13.815107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:13.815549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:13.817011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] PATH_INFO: `/v3/projects/1d44219808dc40dbaedd00b9b003b79b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:13.818702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d44219808dc40dbaedd00b9b003b79b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:13.820753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:13.888727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d44219808dc40dbaedd00b9b003b79b', 'request_id': 'req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.897919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d44219808dc40dbaedd00b9b003b79b', 'request_id': 'req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.906008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d44219808dc40dbaedd00b9b003b79b', 'request_id': 'req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.921920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] The token cache is being invalidate because project 1d44219808dc40dbaedd00b9b003b79b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:13.922200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.922849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:13.924667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1050/2103] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:13 2022] DELETE /identity/v3/projects/1d44219808dc40dbaedd00b9b003b79b => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:13.929519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:13.934738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:13.940493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffb84c6-bd82-406e-8813-abf10771759f service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:14.045669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:14.046321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:14.046648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:14.046929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] PATH_INFO: `/v3/users/fe15214ef55e4f06ad57eb488c81ebbd` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:14.048828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe15214ef55e4f06ad57eb488c81ebbd)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:14.052072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:14.104461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe15214ef55e4f06ad57eb488c81ebbd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.105124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe15214ef55e4f06ad57eb488c81ebbd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.105756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe15214ef55e4f06ad57eb488c81ebbd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.118471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe15214ef55e4f06ad57eb488c81ebbd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.133249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe15214ef55e4f06ad57eb488c81ebbd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.147008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1054/2104] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:13 2022] DELETE /identity/v3/users/fe15214ef55e4f06ad57eb488c81ebbd => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:14.417383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:14.419271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:14.433053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071ee567-e6e7-4d8c-9ba3-91b9308c4fd9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:14.499575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:14.500470 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:14.500723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:14.501017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] PATH_INFO: `/v3/projects/9de663af1e6741529066cd6ef30382f7` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:14.502160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9de663af1e6741529066cd6ef30382f7)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:14.504198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:14.571803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9de663af1e6741529066cd6ef30382f7', 'request_id': 'req-e84f5b7d-e24f-4d27-8085-3942e0ba6335'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.581692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9de663af1e6741529066cd6ef30382f7', 'request_id': 'req-e84f5b7d-e24f-4d27-8085-3942e0ba6335'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.585518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:14.589795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9de663af1e6741529066cd6ef30382f7', 'request_id': 'req-e84f5b7d-e24f-4d27-8085-3942e0ba6335'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.604763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] The token cache is being invalidate because project 9de663af1e6741529066cd6ef30382f7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:14.605131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.605810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e84f5b7d-e24f-4d27-8085-3942e0ba6335 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:14.607430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1051/2105] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:14 2022] DELETE /identity/v3/projects/9de663af1e6741529066cd6ef30382f7 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:17.447233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:17.448825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:17.463146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a78128-888d-47b0-9893-9b7a9cd52368 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:17.525314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:17.525941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:17.526220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:17.526491 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] PATH_INFO: `/v3/users/0ee1005a7ae141749869d4578b59c0da` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:17.527613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0ee1005a7ae141749869d4578b59c0da)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:17.529637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:17.554221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:17.581009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0ee1005a7ae141749869d4578b59c0da'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:17.582448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0ee1005a7ae141749869d4578b59c0da'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:17.582991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0ee1005a7ae141749869d4578b59c0da'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:17.590779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0ee1005a7ae141749869d4578b59c0da'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:17.604104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0ee1005a7ae141749869d4578b59c0da'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:17.626834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1055/2106] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:17 2022] DELETE /identity/v3/users/0ee1005a7ae141749869d4578b59c0da => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:17.632404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ab42a-d873-4100-acb8-da030aaefc73 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:17.632883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ab42a-d873-4100-acb8-da030aaefc73 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:17.633031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260ab42a-d873-4100-acb8-da030aaefc73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:17.660449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-260ab42a-d873-4100-acb8-da030aaefc73 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:17.662216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-260ab42a-d873-4100-acb8-da030aaefc73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:17.725553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1052/2107] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:17.735814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:17.737256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:17.820238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2825a1fd-b211-4455-976d-b43cfe630049 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:17.824073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef63fda-7d72-4c94-91d3-60bbdade0a87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:17.824747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef63fda-7d72-4c94-91d3-60bbdade0a87 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:17.825028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef63fda-7d72-4c94-91d3-60bbdade0a87 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:17.825300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef63fda-7d72-4c94-91d3-60bbdade0a87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:17.836183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef63fda-7d72-4c94-91d3-60bbdade0a87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:17.837759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef63fda-7d72-4c94-91d3-60bbdade0a87 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:17.851093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1056/2108] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:17.981936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453fa678-4d0e-4e2b-a9d5-266bb50d8ad7 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:17.982289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453fa678-4d0e-4e2b-a9d5-266bb50d8ad7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:17.982624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453fa678-4d0e-4e2b-a9d5-266bb50d8ad7 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:17.984465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1053/2109] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:55:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:55:17.997150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:17.998012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:17.998012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:18.045874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:18.047096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:18.125637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1057/2110] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:55:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:18.142021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453fa678-4d0e-4e2b-a9d5-266bb50d8ad7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:18.143786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-453fa678-4d0e-4e2b-a9d5-266bb50d8ad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:18.154469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:18.156212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:18.168064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453fa678-4d0e-4e2b-a9d5-266bb50d8ad7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:18.177043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:18.177692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:18.177972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:18.178242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] PATH_INFO: `/v3/users/dbd7d1a148fd47d9b6f88d8c988182bd` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:18.178366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c811c2b3-83b3-4ff0-af24-16835495bd1e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:18.181195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dbd7d1a148fd47d9b6f88d8c988182bd)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:18.183171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:18.184535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:18.185246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:18.185795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:18.186068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:18.198213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:18.200350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:18.212470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1058/2111] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:18.250318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dbd7d1a148fd47d9b6f88d8c988182bd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.251103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dbd7d1a148fd47d9b6f88d8c988182bd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.260364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dbd7d1a148fd47d9b6f88d8c988182bd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.261156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dbd7d1a148fd47d9b6f88d8c988182bd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.289190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dbd7d1a148fd47d9b6f88d8c988182bd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.314351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:18.315435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1054/2112] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:18 2022] DELETE /identity/v3/users/dbd7d1a148fd47d9b6f88d8c988182bd => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:18.315836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:18.331483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d3ce4d-f4cb-4dab-8db5-dee2bb1cf5d6 service placement] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:18.362298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:18.400291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:18.400938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:18.401301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:18.401741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] PATH_INFO: `/v3/projects/ec57ed444f3d4f9a9b4b79df81681160` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:18.403071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec57ed444f3d4f9a9b4b79df81681160)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:18.405230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:18.458516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec57ed444f3d4f9a9b4b79df81681160', 'request_id': 'req-17d8cf3a-6af5-4788-a402-b1f3c745f76b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.465008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec57ed444f3d4f9a9b4b79df81681160', 'request_id': 'req-17d8cf3a-6af5-4788-a402-b1f3c745f76b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.470815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec57ed444f3d4f9a9b4b79df81681160', 'request_id': 'req-17d8cf3a-6af5-4788-a402-b1f3c745f76b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.485956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] The token cache is being invalidate because project ec57ed444f3d4f9a9b4b79df81681160 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:18.486604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.487375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.489477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1059/2113] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:18 2022] DELETE /identity/v3/projects/ec57ed444f3d4f9a9b4b79df81681160 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:18.493627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:18.495416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:18.509936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa516fc9-408c-4baa-acff-0f8cd7826579 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:18.554020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:18.555536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:18.587248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:18.595562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:18.597019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:18.597432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:18.597823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] PATH_INFO: `/v3/users/bd9872df9cd940fc959364f22bfe04e8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:18.599356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd9872df9cd940fc959364f22bfe04e8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:18.601863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:18.628291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d8cf3a-6af5-4788-a402-b1f3c745f76b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:18.634487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:18.635197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:18.635530 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:18.635825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] PATH_INFO: `/v3/projects/3f342e4529b64add8e216082ef9dff14` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:18.637038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3f342e4529b64add8e216082ef9dff14)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:18.639090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:18.651203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd9872df9cd940fc959364f22bfe04e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.651818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd9872df9cd940fc959364f22bfe04e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.658827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd9872df9cd940fc959364f22bfe04e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.659477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd9872df9cd940fc959364f22bfe04e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.673722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd9872df9cd940fc959364f22bfe04e8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.690616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1055/2114] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:18 2022] DELETE /identity/v3/users/bd9872df9cd940fc959364f22bfe04e8 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:18.691632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3f342e4529b64add8e216082ef9dff14', 'request_id': 'req-32466b7c-9ccb-46c3-867c-85f15396f9a3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.698363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3f342e4529b64add8e216082ef9dff14', 'request_id': 'req-32466b7c-9ccb-46c3-867c-85f15396f9a3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.702239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:18.704715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3f342e4529b64add8e216082ef9dff14', 'request_id': 'req-32466b7c-9ccb-46c3-867c-85f15396f9a3'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.719170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] The token cache is being invalidate because project 3f342e4529b64add8e216082ef9dff14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:18.719758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.720155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.721670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1060/2115] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:18 2022] DELETE /identity/v3/projects/3f342e4529b64add8e216082ef9dff14 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:18.727265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:18.729422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:18.807867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-645b00f9-349c-419d-8fcb-a5d75e072059 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:18.814366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:18.815066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:18.815410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:18.815743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] PATH_INFO: `/v3/users/0a1cc0e49fd8471abb2f3a4bd0ef0541` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:18.817263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a1cc0e49fd8471abb2f3a4bd0ef0541)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:18.819308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:18.871505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a1cc0e49fd8471abb2f3a4bd0ef0541'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.872214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a1cc0e49fd8471abb2f3a4bd0ef0541'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.879332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a1cc0e49fd8471abb2f3a4bd0ef0541'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.880376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a1cc0e49fd8471abb2f3a4bd0ef0541'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.893713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a1cc0e49fd8471abb2f3a4bd0ef0541'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:18.897921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:18.909269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1056/2116] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:18 2022] DELETE /identity/v3/users/0a1cc0e49fd8471abb2f3a4bd0ef0541 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:18.987293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:18.989018 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.062083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32466b7c-9ccb-46c3-867c-85f15396f9a3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.094987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.096462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.127270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.128114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.128282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.128601 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] PATH_INFO: `/v3/projects/17214a73de22428eb996f719faad30fc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.130129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17214a73de22428eb996f719faad30fc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.132072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.178610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36c2406f-34cd-4366-abfb-c9983962fd21 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.180918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17214a73de22428eb996f719faad30fc', 'request_id': 'req-9aba1200-59c5-4c15-b5bc-3227a0d336c4'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.188015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17214a73de22428eb996f719faad30fc', 'request_id': 'req-9aba1200-59c5-4c15-b5bc-3227a0d336c4'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.194770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17214a73de22428eb996f719faad30fc', 'request_id': 'req-9aba1200-59c5-4c15-b5bc-3227a0d336c4'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.207302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] The token cache is being invalidate because project 17214a73de22428eb996f719faad30fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:19.207608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.208242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9aba1200-59c5-4c15-b5bc-3227a0d336c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.209754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1061/2117] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:18 2022] DELETE /identity/v3/projects/17214a73de22428eb996f719faad30fc => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:19.214356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d030da5-29fb-409e-b781-ad99d3e920dd None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.214643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d030da5-29fb-409e-b781-ad99d3e920dd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.214911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d030da5-29fb-409e-b781-ad99d3e920dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.235025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4d030da5-29fb-409e-b781-ad99d3e920dd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:19.236282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4d030da5-29fb-409e-b781-ad99d3e920dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.237553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.238925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.239275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.239882 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] PATH_INFO: `/v3/projects/1fa5587735f64d3d95b425da75a1350d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.240841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fa5587735f64d3d95b425da75a1350d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.242929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.246324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1062/2118] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:19.255501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6b319-ddc5-4caa-85b5-e368ee9a6fd2 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.255786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6b319-ddc5-4caa-85b5-e368ee9a6fd2 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.256090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6b319-ddc5-4caa-85b5-e368ee9a6fd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.275885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-91e6b319-ddc5-4caa-85b5-e368ee9a6fd2 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:19.277159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-91e6b319-ddc5-4caa-85b5-e368ee9a6fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.285849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1063/2119] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:19.289758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6b319-ddc5-4caa-85b5-e368ee9a6fd2 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.293564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fa5587735f64d3d95b425da75a1350d', 'request_id': 'req-d287110c-c72d-43a3-bcec-9e777ebe4786'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.299562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6b319-ddc5-4caa-85b5-e368ee9a6fd2 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.304790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fa5587735f64d3d95b425da75a1350d', 'request_id': 'req-d287110c-c72d-43a3-bcec-9e777ebe4786'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.311867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fa5587735f64d3d95b425da75a1350d', 'request_id': 'req-d287110c-c72d-43a3-bcec-9e777ebe4786'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.312582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:DELETE=9,keystone:SELECT=89,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:19.327557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] The token cache is being invalidate because project 1fa5587735f64d3d95b425da75a1350d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:19.328135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.328918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d287110c-c72d-43a3-bcec-9e777ebe4786 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:19.330306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1057/2120] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:19 2022] DELETE /identity/v3/projects/1fa5587735f64d3d95b425da75a1350d => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:19.335631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c57a06f-9848-421a-bf92-323dbbc630fa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.335631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c57a06f-9848-421a-bf92-323dbbc630fa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.336103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c57a06f-9848-421a-bf92-323dbbc630fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.357103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6c57a06f-9848-421a-bf92-323dbbc630fa None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:19.358404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6c57a06f-9848-421a-bf92-323dbbc630fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.379180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.380361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.380820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.381235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.382031 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:19.382493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.384517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.394967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:19.401814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1064/2121] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:19.406886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.408689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.422201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d4a79b-254f-460f-a3d6-23c3c6ab7940 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.427804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.428434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.428772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.428985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.429664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.432951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.436053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1058/2122] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:19.447441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ed6fb-fd5e-42b1-a60e-d4d74f5b9b2e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.447729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ed6fb-fd5e-42b1-a60e-d4d74f5b9b2e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.448018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ed6fb-fd5e-42b1-a60e-d4d74f5b9b2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.470286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fa7ed6fb-fd5e-42b1-a60e-d4d74f5b9b2e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:19.471505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fa7ed6fb-fd5e-42b1-a60e-d4d74f5b9b2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.483975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1059/2123] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:19.489682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ed6fb-fd5e-42b1-a60e-d4d74f5b9b2e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.504184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ed6fb-fd5e-42b1-a60e-d4d74f5b9b2e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.508239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1065/2124] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/projects => generated 375 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:19.511707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.512288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.512497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.512846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.513184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.513905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:19.514309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.516108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.530793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1060/2125] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:19.536812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.553749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09b9c0c-b700-4627-9bdc-e234e3513b5c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.586245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06e549b-f629-4774-9ef7-934de95916d2 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.593194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.594388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.594675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.595013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.598195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.602275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.624923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.625701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.626036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.626464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.627163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.630501 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.637674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1066/2126] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/users => generated 288 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:19.645575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.656351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95a4464-b7e0-433f-b2ad-6e9ed95b86c6 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.661341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.661926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.662201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.662468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.663967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1061/2127] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/projects => generated 377 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:19.664445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.664770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.666821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.680041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1067/2128] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:19 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:19.681245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.692845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b6cb34-8a5a-44d0-8a9c-ad68b3e7e9a9 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.730286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be647c9-f3e8-4440-b0e8-4abf84ac1998 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.735420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.736455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.736729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.736995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.737617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.740842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.751791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.752436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.752715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.753108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.754893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.756810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.758924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:19.763823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1062/2129] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/users => generated 289 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:19.768112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.771429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1068/2130] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:19 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:19.772283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.779202 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7226208-1d70-4865-9aae-80f2051c5340 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.781888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917a96a3-4bc8-40ce-8023-90d20a8f9fdd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.783824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.784445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.784763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.785036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.785667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.786722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.787237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.787371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.787617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.787887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] PATH_INFO: `/v3/projects/c240f6399e944b2585f2b24b609d4411/users/91a3b1df4491416ea6152d85dc766cec/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.797524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1063/2131] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:19.802102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.810090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c240f6399e944b2585f2b24b609d4411, user_id=91a3b1df4491416ea6152d85dc766cec, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.812898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4822206-0196-4002-9b5f-98d6e5a7d3a5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.813008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.819168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcb0b29-42f7-4d33-8a83-7d489255b793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.819891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcb0b29-42f7-4d33-8a83-7d489255b793 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.820233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcb0b29-42f7-4d33-8a83-7d489255b793 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.820531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcb0b29-42f7-4d33-8a83-7d489255b793 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.821199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcb0b29-42f7-4d33-8a83-7d489255b793 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.823457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcb0b29-42f7-4d33-8a83-7d489255b793 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.827578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1069/2132] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:19 2022] PUT /identity/v3/projects/c240f6399e944b2585f2b24b609d4411/users/91a3b1df4491416ea6152d85dc766cec/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:19.834609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1064/2133] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:19.838727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.839215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d09854-6898-425e-a8e8-cfe8eb8605c7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.839528 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d09854-6898-425e-a8e8-cfe8eb8605c7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.839824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d09854-6898-425e-a8e8-cfe8eb8605c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.853697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:19.870673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d5d09854-6898-425e-a8e8-cfe8eb8605c7 None None] MFA Rules not processed for user `91a3b1df4491416ea6152d85dc766cec`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:19.872434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d5d09854-6898-425e-a8e8-cfe8eb8605c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:19.903356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552efca1-c09f-484c-bb94-51f7a40f7308 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.908123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39339112-eb13-4645-abcd-67addfc3b214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:19.908772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39339112-eb13-4645-abcd-67addfc3b214 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.909046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39339112-eb13-4645-abcd-67addfc3b214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.909311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39339112-eb13-4645-abcd-67addfc3b214 admin admin] PATH_INFO: `/v3/projects/606596b4dfad42ecae365c057b989f19/users/efa27d88116c4f2595678a0ca800697e/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.925247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39339112-eb13-4645-abcd-67addfc3b214 admin admin] RBAC: Authorizing `identity:create_grant(project_id=606596b4dfad42ecae365c057b989f19, user_id=efa27d88116c4f2595678a0ca800697e, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:19.928145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39339112-eb13-4645-abcd-67addfc3b214 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:19.939919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1065/2134] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:19.941990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1070/2135] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:19 2022] PUT /identity/v3/projects/606596b4dfad42ecae365c057b989f19/users/efa27d88116c4f2595678a0ca800697e/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:19.945480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d09854-6898-425e-a8e8-cfe8eb8605c7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:19.952053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6db7b-fbed-46fa-a5b0-6b25e70cd6f0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:19.952341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6db7b-fbed-46fa-a5b0-6b25e70cd6f0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:19.952614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb6db7b-fbed-46fa-a5b0-6b25e70cd6f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:19.955759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d09854-6898-425e-a8e8-cfe8eb8605c7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:19.980906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1fb6db7b-fbed-46fa-a5b0-6b25e70cd6f0 None None] MFA Rules not processed for user `efa27d88116c4f2595678a0ca800697e`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:19.982118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1fb6db7b-fbed-46fa-a5b0-6b25e70cd6f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:20.017802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.018647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.019134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.019975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.020657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.022560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.050054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1066/2136] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/projects => generated 373 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:20.051837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1071/2137] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:19 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:20.056172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb6db7b-fbed-46fa-a5b0-6b25e70cd6f0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:20.063533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:20.065203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:20.068153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb6db7b-fbed-46fa-a5b0-6b25e70cd6f0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:20.113360 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:20.130178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.130786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.131062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.131334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.132024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.134002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.148014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2db6e-3060-47f4-88ae-a607dcb5b657 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:20.153154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.154028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.154478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.154868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.159211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1072/2138] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:20 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:20.159971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:20.171004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61bf1fdc-9c27-4c8d-a1c2-a4e16b1c3a9d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:20.171557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:20.175869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.176509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.176790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.177064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.177728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.179371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.192498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1073/2139] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:20.196496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:20.206471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e52210a-c17c-4395-9a3d-deca160c2290 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:20.212556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.213311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.213797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.214170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] PATH_INFO: `/v3/projects/0d16ea5503bb4118b4ec4dacf2137964/users/2e5e8d8fe64a46eb9ef9b2e0769cc7ec/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.223115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.224680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.229248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d16ea5503bb4118b4ec4dacf2137964, user_id=2e5e8d8fe64a46eb9ef9b2e0769cc7ec, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.231877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.233777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1067/2140] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:20 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:20.245858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1074/2141] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:20 2022] PUT /identity/v3/projects/0d16ea5503bb4118b4ec4dacf2137964/users/2e5e8d8fe64a46eb9ef9b2e0769cc7ec/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:20.249703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:20.264438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f10de21-04b1-4be8-bffc-7bc90384112e service glance] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:20.326476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e58168-7a61-4915-b6ec-bd6d8d48dd68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.327086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e58168-7a61-4915-b6ec-bd6d8d48dd68 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.327393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e58168-7a61-4915-b6ec-bd6d8d48dd68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.327692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e58168-7a61-4915-b6ec-bd6d8d48dd68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.328374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e58168-7a61-4915-b6ec-bd6d8d48dd68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.329995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e58168-7a61-4915-b6ec-bd6d8d48dd68 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.341295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1068/2142] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:20 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:20.345340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:20.356383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6be365-c6a1-432a-af06-2ac95a51d6ce admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:20.361749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.362456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.362736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.363529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] PATH_INFO: `/v3/projects/0d16ea5503bb4118b4ec4dacf2137964/users/2e5e8d8fe64a46eb9ef9b2e0769cc7ec/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.365596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d16ea5503bb4118b4ec4dacf2137964, user_id=2e5e8d8fe64a46eb9ef9b2e0769cc7ec, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.368444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.380800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1075/2143] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:20 2022] PUT /identity/v3/projects/0d16ea5503bb4118b4ec4dacf2137964/users/2e5e8d8fe64a46eb9ef9b2e0769cc7ec/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:20.391053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6fdd1-5e66-47f5-a3ef-454cfcb780d3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.391341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6fdd1-5e66-47f5-a3ef-454cfcb780d3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.391613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6fdd1-5e66-47f5-a3ef-454cfcb780d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.419572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-57c6fdd1-5e66-47f5-a3ef-454cfcb780d3 None None] MFA Rules not processed for user `2e5e8d8fe64a46eb9ef9b2e0769cc7ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:20.420824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-57c6fdd1-5e66-47f5-a3ef-454cfcb780d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:20.496866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1069/2144] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:20 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:20.508025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:20.509935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:20.575372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32164d22-cbc8-4272-9161-567deb0aef97 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:20.580056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e5db0b-bab5-4cc7-b56f-a5430e01e5bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:20.581223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5db0b-bab5-4cc7-b56f-a5430e01e5bd service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:20.581223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5db0b-bab5-4cc7-b56f-a5430e01e5bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:20.581501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e5db0b-bab5-4cc7-b56f-a5430e01e5bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:20.654429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5db0b-bab5-4cc7-b56f-a5430e01e5bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:20.656674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e5db0b-bab5-4cc7-b56f-a5430e01e5bd service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:20.679201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1076/2145] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:20 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:23.343720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6fdd1-5e66-47f5-a3ef-454cfcb780d3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:23.387463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:23.409075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6fdd1-5e66-47f5-a3ef-454cfcb780d3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:23.413809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:23.414445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:23.414718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:23.414981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] PATH_INFO: `/v3/users/91a3b1df4491416ea6152d85dc766cec` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:23.416105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91a3b1df4491416ea6152d85dc766cec)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:23.418029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:23.467862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91a3b1df4491416ea6152d85dc766cec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:23.468542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91a3b1df4491416ea6152d85dc766cec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:23.475325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91a3b1df4491416ea6152d85dc766cec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:23.475972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91a3b1df4491416ea6152d85dc766cec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:23.490026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91a3b1df4491416ea6152d85dc766cec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:23.502874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1070/2146] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:23 2022] DELETE /identity/v3/users/91a3b1df4491416ea6152d85dc766cec => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:23.508647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:23.509116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:23.509204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:23.528095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:23.529297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:23.569521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:23.591281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1077/2147] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:23.601182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:23.602569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:23.664398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbf872-cbf1-4c23-a00c-074b2c3c4803 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:23.668183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:23.668817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:23.669097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:23.669367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:23.678661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:23.680188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:23.690077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1071/2148] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:23.973374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:23.974882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:23.997070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2e4233-1bcb-4095-a5b3-7aed2125cb54 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:24.003185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:24.004698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:24.005270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:24.005534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] PATH_INFO: `/v3/projects/c240f6399e944b2585f2b24b609d4411` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:24.006690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c240f6399e944b2585f2b24b609d4411)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:24.008747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:24.060299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c240f6399e944b2585f2b24b609d4411', 'request_id': 'req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.068237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c240f6399e944b2585f2b24b609d4411', 'request_id': 'req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.074059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c240f6399e944b2585f2b24b609d4411', 'request_id': 'req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.091246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] The token cache is being invalidate because project c240f6399e944b2585f2b24b609d4411 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:24.091542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.092324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.093734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1078/2149] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:23 2022] DELETE /identity/v3/projects/c240f6399e944b2585f2b24b609d4411 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:24.098311 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:24.108594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be3b14-0aac-401d-a76f-df3573abb740 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:24.176909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:24.177542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:24.177883 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:24.178067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] PATH_INFO: `/v3/users/2e5e8d8fe64a46eb9ef9b2e0769cc7ec` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:24.179249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e5e8d8fe64a46eb9ef9b2e0769cc7ec)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:24.181250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:24.182976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:24.238804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e5e8d8fe64a46eb9ef9b2e0769cc7ec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.239468 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e5e8d8fe64a46eb9ef9b2e0769cc7ec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.268522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e5e8d8fe64a46eb9ef9b2e0769cc7ec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.269854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e5e8d8fe64a46eb9ef9b2e0769cc7ec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.288515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3aa39870-2388-4121-9f32-ed49ae53d4f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e5e8d8fe64a46eb9ef9b2e0769cc7ec'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.302590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1072/2150] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:24 2022] DELETE /identity/v3/users/2e5e8d8fe64a46eb9ef9b2e0769cc7ec => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:24.613400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:24.615082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:24.629809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2bf02e-d9cc-4308-b88f-99dfc51f133a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:24.688037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:24.688677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:24.688954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:24.689229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] PATH_INFO: `/v3/projects/0d16ea5503bb4118b4ec4dacf2137964` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:24.690347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d16ea5503bb4118b4ec4dacf2137964)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:24.693193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:24.740709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d16ea5503bb4118b4ec4dacf2137964', 'request_id': 'req-1a106ff4-942e-4c05-9dcf-43734972e56a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.747245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d16ea5503bb4118b4ec4dacf2137964', 'request_id': 'req-1a106ff4-942e-4c05-9dcf-43734972e56a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.752756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d16ea5503bb4118b4ec4dacf2137964', 'request_id': 'req-1a106ff4-942e-4c05-9dcf-43734972e56a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.757357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:24.769667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] The token cache is being invalidate because project 0d16ea5503bb4118b4ec4dacf2137964 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:24.770728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.771365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1a106ff4-942e-4c05-9dcf-43734972e56a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:24.773704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1079/2151] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:24 2022] DELETE /identity/v3/projects/0d16ea5503bb4118b4ec4dacf2137964 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:24.778754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ddbd96-e934-419e-be28-28642fad298f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:24.779347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ddbd96-e934-419e-be28-28642fad298f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:24.779347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ddbd96-e934-419e-be28-28642fad298f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:24.801912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-91ddbd96-e934-419e-be28-28642fad298f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:24.803212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-91ddbd96-e934-419e-be28-28642fad298f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:24.865963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1073/2152] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:24.874404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349d05f-c494-4df6-957d-444d4bcbc42b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:24.874701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349d05f-c494-4df6-957d-444d4bcbc42b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:24.874965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1349d05f-c494-4df6-957d-444d4bcbc42b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:24.895309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1349d05f-c494-4df6-957d-444d4bcbc42b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:24.896719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1349d05f-c494-4df6-957d-444d4bcbc42b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:24.904921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1080/2153] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:24.909255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ddbd96-e934-419e-be28-28642fad298f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:24.919237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ddbd96-e934-419e-be28-28642fad298f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:24.923750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:24.924385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:24.924666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:24.924930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:24.926363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:24.926699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:24.928387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:24.939567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1074/2154] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:24.944455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349d05f-c494-4df6-957d-444d4bcbc42b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:24.954079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1349d05f-c494-4df6-957d-444d4bcbc42b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:24.959002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:24.959610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:24.959887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:24.960190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:24.960848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:24.962773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:24.995536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1081/2155] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:24 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:25.000044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:25.012194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44285eed-32f2-4182-9d99-b933cc13a006 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:25.075292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:25.075891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:25.076209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:25.076475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:25.077131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:25.079050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:25.090275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:25.101157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1075/2156] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:24 2022] POST /identity/v3/users => generated 287 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:25.105440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:25.116593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a820a01-fe05-4d73-9bf7-ddd5a3f1ebe0 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:25.121152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:25.121747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:25.122021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:25.122285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:25.122915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:25.124545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:25.137509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1082/2157] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:25.141445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:25.154030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ab1ba6-aa31-4301-aee5-39fb1db29190 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:25.161003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07c02a2-4f18-4043-a394-b595302924f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:25.161605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c02a2-4f18-4043-a394-b595302924f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:25.161889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c02a2-4f18-4043-a394-b595302924f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:25.162161 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c02a2-4f18-4043-a394-b595302924f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:25.162808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07c02a2-4f18-4043-a394-b595302924f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:25.164485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07c02a2-4f18-4043-a394-b595302924f6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:25.176224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1076/2158] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:25 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:25.180577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:25.191862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f013ee88-c97d-4858-8d42-ee8a5834ccca admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:25.196596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:25.197252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:25.197533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:25.197803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] PATH_INFO: `/v3/projects/130b2f7474ad4c8d8f834c5cb4dbe636/users/8871e9e4e5f64a96b083b61db4c5264d/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:25.212468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] RBAC: Authorizing `identity:create_grant(project_id=130b2f7474ad4c8d8f834c5cb4dbe636, user_id=8871e9e4e5f64a96b083b61db4c5264d, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:25.215112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:25.229124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1083/2159] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:25 2022] PUT /identity/v3/projects/130b2f7474ad4c8d8f834c5cb4dbe636/users/8871e9e4e5f64a96b083b61db4c5264d/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:25.239655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8591663d-525f-4c35-89f5-1f4ec1a8b703 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:25.239952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8591663d-525f-4c35-89f5-1f4ec1a8b703 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:25.240268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8591663d-525f-4c35-89f5-1f4ec1a8b703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:25.269636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8591663d-525f-4c35-89f5-1f4ec1a8b703 None None] MFA Rules not processed for user `8871e9e4e5f64a96b083b61db4c5264d`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:25.270871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8591663d-525f-4c35-89f5-1f4ec1a8b703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:25.338636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1077/2160] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:25.349669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:25.351329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:25.409479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d4e189-6aa3-4281-be65-6b83d04c1daf admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:25.413405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbdf1da-696e-45d6-b305-df7f1ffc3e4d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:25.414016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbdf1da-696e-45d6-b305-df7f1ffc3e4d service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:25.414295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbdf1da-696e-45d6-b305-df7f1ffc3e4d service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:25.414563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbdf1da-696e-45d6-b305-df7f1ffc3e4d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:25.423487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbdf1da-696e-45d6-b305-df7f1ffc3e4d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:25.425211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbdf1da-696e-45d6-b305-df7f1ffc3e4d service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:25.435012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1084/2161] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:25 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:28.619393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8591663d-525f-4c35-89f5-1f4ec1a8b703 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:28.716625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8591663d-525f-4c35-89f5-1f4ec1a8b703 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:28.724764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:28.725548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:28.725916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:28.726193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] PATH_INFO: `/v3/users/8871e9e4e5f64a96b083b61db4c5264d` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:28.727515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8871e9e4e5f64a96b083b61db4c5264d)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:28.729585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:28.794280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8871e9e4e5f64a96b083b61db4c5264d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:28.795678 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8871e9e4e5f64a96b083b61db4c5264d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:28.811223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8871e9e4e5f64a96b083b61db4c5264d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:28.812392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8871e9e4e5f64a96b083b61db4c5264d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:28.829244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8871e9e4e5f64a96b083b61db4c5264d'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:28.833496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:28.864179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1078/2162] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:28 2022] DELETE /identity/v3/users/8871e9e4e5f64a96b083b61db4c5264d => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:28.866308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:28.866813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:28.866894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:28.892572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:28.902839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:28.988132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:28.990065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:29.010797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1085/2163] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:29.021169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:29.023263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:29.083470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:29.094010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4d2a43-2c29-49f5-9fda-b47226e573c4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:29.100070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:29.100708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.100983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.101253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] PATH_INFO: `/v3/users/a39440ed016445a7bfc7ac7fba6661f2` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.102374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a39440ed016445a7bfc7ac7fba6661f2)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:29.104400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:29.107842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0c0fb1-f55a-4d76-9196-b031a952f0ab None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:29.111786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:29.113267 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.113739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.114212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.126867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:29.128566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:29.143187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1086/2164] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:29.155178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a39440ed016445a7bfc7ac7fba6661f2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.155889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a39440ed016445a7bfc7ac7fba6661f2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.166718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a39440ed016445a7bfc7ac7fba6661f2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.167471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a39440ed016445a7bfc7ac7fba6661f2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.180671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa01f97-bee8-4c75-a80a-d60b97beb39d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a39440ed016445a7bfc7ac7fba6661f2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.193830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1079/2165] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:28 2022] DELETE /identity/v3/users/a39440ed016445a7bfc7ac7fba6661f2 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:29.467394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:29.468847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:29.486844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd86272a-9576-42e8-9349-d9391993bbef service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:29.561775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:29.567822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.568343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.568706 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] PATH_INFO: `/v3/projects/130b2f7474ad4c8d8f834c5cb4dbe636` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.570191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=130b2f7474ad4c8d8f834c5cb4dbe636)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:29.573100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:29.633794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '130b2f7474ad4c8d8f834c5cb4dbe636', 'request_id': 'req-6fff61b9-7d26-40e8-934d-d57b58e047f8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.642965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '130b2f7474ad4c8d8f834c5cb4dbe636', 'request_id': 'req-6fff61b9-7d26-40e8-934d-d57b58e047f8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.649562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '130b2f7474ad4c8d8f834c5cb4dbe636', 'request_id': 'req-6fff61b9-7d26-40e8-934d-d57b58e047f8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.664459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] The token cache is being invalidate because project 130b2f7474ad4c8d8f834c5cb4dbe636 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:29.665212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.666118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.667906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1087/2166] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:29 2022] DELETE /identity/v3/projects/130b2f7474ad4c8d8f834c5cb4dbe636 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:29.674344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bf621a-18a0-423d-8d19-72dcc3f1847d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.674810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bf621a-18a0-423d-8d19-72dcc3f1847d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.675624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bf621a-18a0-423d-8d19-72dcc3f1847d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.696363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f8bf621a-18a0-423d-8d19-72dcc3f1847d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:29.697692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f8bf621a-18a0-423d-8d19-72dcc3f1847d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:29.753666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:29.756002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:29.813879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1080/2167] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:29.822698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553c6c4-7cc0-4d27-a8ea-de2455d5e7df None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.823067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553c6c4-7cc0-4d27-a8ea-de2455d5e7df None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.823409 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553c6c4-7cc0-4d27-a8ea-de2455d5e7df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.838579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff61b9-7d26-40e8-934d-d57b58e047f8 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:29.843302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:29.843930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.844232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.844329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e553c6c4-7cc0-4d27-a8ea-de2455d5e7df None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:29.844506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] PATH_INFO: `/v3/projects/477dd0575c8741b0815ddf5f22297db3` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.845655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] RBAC: Authorizing `identity:delete_project(project_id=477dd0575c8741b0815ddf5f22297db3)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:29.845791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e553c6c4-7cc0-4d27-a8ea-de2455d5e7df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:29.847660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:29.855855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1081/2168] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:29.859925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e553c6c4-7cc0-4d27-a8ea-de2455d5e7df None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:29.862193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:29.869851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e553c6c4-7cc0-4d27-a8ea-de2455d5e7df None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:29.875018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:29.875609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.876031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.876319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.877106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:29.877400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:29.879829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:29.881552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:29.895778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1082/2169] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:29.900054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:29.910867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42b60e-9098-4ad9-9665-ef39d236f0a7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:29.911599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '477dd0575c8741b0815ddf5f22297db3', 'request_id': 'req-a2a24271-7642-4aaf-b7ae-01c241ee0483'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.918174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '477dd0575c8741b0815ddf5f22297db3', 'request_id': 'req-a2a24271-7642-4aaf-b7ae-01c241ee0483'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.924098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '477dd0575c8741b0815ddf5f22297db3', 'request_id': 'req-a2a24271-7642-4aaf-b7ae-01c241ee0483'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.938057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] The token cache is being invalidate because project 477dd0575c8741b0815ddf5f22297db3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:29.938345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.939015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a2a24271-7642-4aaf-b7ae-01c241ee0483 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:29.941201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1088/2170] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:29 2022] DELETE /identity/v3/projects/477dd0575c8741b0815ddf5f22297db3 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:29.946029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d78ff1-28f6-4118-be69-363cc81e25e8 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.946358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d78ff1-28f6-4118-be69-363cc81e25e8 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.946636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d78ff1-28f6-4118-be69-363cc81e25e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.967103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-08d78ff1-28f6-4118-be69-363cc81e25e8 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:29.968329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-08d78ff1-28f6-4118-be69-363cc81e25e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:29.975159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:29.975763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:29.976067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:29.976337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:29.977031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:29.978890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.015486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1083/2171] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:29 2022] POST /identity/v3/projects => generated 373 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:30.021603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.023355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:30.034747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1089/2172] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:30.042171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cdfd02-9922-4e3c-a380-f9c3e48b39bd None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.042457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cdfd02-9922-4e3c-a380-f9c3e48b39bd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.042726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cdfd02-9922-4e3c-a380-f9c3e48b39bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.065800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-66cdfd02-9922-4e3c-a380-f9c3e48b39bd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:30.067380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-66cdfd02-9922-4e3c-a380-f9c3e48b39bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:30.092423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93862fdd-2c1d-4aa1-9318-9123dcf3073c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.097773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.098559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.098848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.099145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.099766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.101724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.124580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1084/2173] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:30 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:30.128610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.134333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1090/2174] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:30.138373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cdfd02-9922-4e3c-a380-f9c3e48b39bd None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.139033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1287b6-069c-4a8d-944f-00af591cb3da admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.144392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.145109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.145374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.145633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.146517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.148116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.150137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cdfd02-9922-4e3c-a380-f9c3e48b39bd None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.155899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.156527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.156995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.157268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.157912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:30.158277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.161708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1085/2175] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:30.161863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.162764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.172446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40585b7-9a2f-4cfd-be81-6bc3a81223e0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.172899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1091/2176] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:30.177867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.181051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.181673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.182003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.182283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.182922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.184587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.188150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd6f4a-d515-4ebd-9167-fe8a9c99808f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.193364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.194060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.194343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.194742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.195290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.196158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1086/2177] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:30.197542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.200206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.210157 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4855f28d-671d-4627-965f-53136f6c4646 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.214816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d087e2b6-059c-4f2b-b93b-36e8acccb6d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.215497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d087e2b6-059c-4f2b-b93b-36e8acccb6d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.215773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d087e2b6-059c-4f2b-b93b-36e8acccb6d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.216067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d087e2b6-059c-4f2b-b93b-36e8acccb6d0 admin admin] PATH_INFO: `/v3/projects/3a0380c941f644a794f51e1f575faf00/users/47a422989ce94168a042135234bea46b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.229876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1092/2178] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:30 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:30.231252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.231382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d087e2b6-059c-4f2b-b93b-36e8acccb6d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a0380c941f644a794f51e1f575faf00, user_id=47a422989ce94168a042135234bea46b, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.234176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d087e2b6-059c-4f2b-b93b-36e8acccb6d0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.236622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:30.255631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1087/2179] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:30 2022] PUT /identity/v3/projects/3a0380c941f644a794f51e1f575faf00/users/47a422989ce94168a042135234bea46b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:30.260563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.260850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.261120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.280023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:30.281227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:30.298644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:30.303921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c10abd0-41ba-4b75-976e-10f6fc2a64ce admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.346983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1088/2180] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:30.356445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.358193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:30.375820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.376684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.376910 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.377253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.378143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.381144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.403637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1093/2181] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:30 2022] POST /identity/v3/users => generated 287 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:30.408805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.421897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ee884c-4eb9-4385-8151-f8d524caf8d3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.428126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99c0301-04b0-449f-9019-9a1058f856a3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.430327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.431205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.431482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.431734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.431855 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.432366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.432478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.432641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.432906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.434209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.442831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.444283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.446366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1094/2182] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:30.450309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.453862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1089/2183] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:55:30.461449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb0c56-6f63-4173-98b2-6bc32387cb24 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.466532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3cace9-6442-40fa-bcc8-ea60af60450e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.467164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3cace9-6442-40fa-bcc8-ea60af60450e admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.467455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3cace9-6442-40fa-bcc8-ea60af60450e admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.467697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3cace9-6442-40fa-bcc8-ea60af60450e admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.468584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3cace9-6442-40fa-bcc8-ea60af60450e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.470332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3cace9-6442-40fa-bcc8-ea60af60450e admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.483806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1095/2184] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:30.488977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.498928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ada3f0-d98e-4fe5-8883-bd2ed1045b43 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.504017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.505139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.505656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.505935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] PATH_INFO: `/v3/projects/7d25c483d30d4ac0b8818e5e6bdaaae3/users/4a8ad2ec844a4eecbcaa442055762562/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.523335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7d25c483d30d4ac0b8818e5e6bdaaae3, user_id=4a8ad2ec844a4eecbcaa442055762562, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.526103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.541028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1090/2185] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:30 2022] PUT /identity/v3/projects/7d25c483d30d4ac0b8818e5e6bdaaae3/users/4a8ad2ec844a4eecbcaa442055762562/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:30.551892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1dab5d-9cd6-4bf8-a7dd-1123cc4b370c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.552225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1dab5d-9cd6-4bf8-a7dd-1123cc4b370c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.552505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1dab5d-9cd6-4bf8-a7dd-1123cc4b370c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.583044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8b1dab5d-9cd6-4bf8-a7dd-1123cc4b370c None None] MFA Rules not processed for user `4a8ad2ec844a4eecbcaa442055762562`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:30.584412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8b1dab5d-9cd6-4bf8-a7dd-1123cc4b370c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:30.660411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1096/2186] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:30 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:30.672074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:30.674159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:30.760821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2284df-6fbf-4842-948c-508857a20f1b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:30.766705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:30.767627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:30.768110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:30.768450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:30.794007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:30.802353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:30.811758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1091/2187] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:31.236356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316686fc-e6da-4da9-802d-868d3e8f10a1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:31.236884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316686fc-e6da-4da9-802d-868d3e8f10a1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:31.236884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316686fc-e6da-4da9-802d-868d3e8f10a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:31.257902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:31.268003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-316686fc-e6da-4da9-802d-868d3e8f10a1 None None] MFA Rules not processed for user `47a422989ce94168a042135234bea46b`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:31.269820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-316686fc-e6da-4da9-802d-868d3e8f10a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:31.339563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1097/2188] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:31.350443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:31.351989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:31.389059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:31.423814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe99da-9272-4b81-8fc3-40490e69c6ef service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:31.428770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb08999b-908e-4765-b3f8-1b819c2c28b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:31.440178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb08999b-908e-4765-b3f8-1b819c2c28b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:31.440178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb08999b-908e-4765-b3f8-1b819c2c28b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:31.440178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb08999b-908e-4765-b3f8-1b819c2c28b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:31.440178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb08999b-908e-4765-b3f8-1b819c2c28b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:31.440687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb08999b-908e-4765-b3f8-1b819c2c28b1 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:31.449679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1092/2189] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:31 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:33.563890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316686fc-e6da-4da9-802d-868d3e8f10a1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:33.626484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316686fc-e6da-4da9-802d-868d3e8f10a1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:33.631594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:33.632252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:33.632528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:33.632794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] PATH_INFO: `/v3/users/4a8ad2ec844a4eecbcaa442055762562` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:33.633899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a8ad2ec844a4eecbcaa442055762562)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:33.635779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:33.685734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a8ad2ec844a4eecbcaa442055762562'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:33.686382 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a8ad2ec844a4eecbcaa442055762562'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:33.686988 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a8ad2ec844a4eecbcaa442055762562'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:33.694104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a8ad2ec844a4eecbcaa442055762562'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:33.705804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a8ad2ec844a4eecbcaa442055762562'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:33.718712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1098/2190] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:33 2022] DELETE /identity/v3/users/4a8ad2ec844a4eecbcaa442055762562 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:33.723913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f616e-f524-428e-9d50-a1deb6606744 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:33.724393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f616e-f524-428e-9d50-a1deb6606744 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:33.724500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-324f616e-f524-428e-9d50-a1deb6606744 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:33.744955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-324f616e-f524-428e-9d50-a1deb6606744 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:33.746150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-324f616e-f524-428e-9d50-a1deb6606744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:33.809300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1093/2191] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:33.820496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:33.821906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:33.842511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:33.894204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ee89ea-60ea-4f5c-b07b-30c085a000e7 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:33.894893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bc57fd-5d92-45f0-a63f-58a9700200fe None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:33.895283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bc57fd-5d92-45f0-a63f-58a9700200fe None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:33.895485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bc57fd-5d92-45f0-a63f-58a9700200fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:33.897923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:33.898530 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:33.898814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:33.899067 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:33.908198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:33.909615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:33.915077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c0bc57fd-5d92-45f0-a63f-58a9700200fe None None] MFA Rules not processed for user `47a422989ce94168a042135234bea46b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:33.916407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c0bc57fd-5d92-45f0-a63f-58a9700200fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:33.919158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1099/2192] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:33.979668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1094/2193] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:34.245707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:34.246197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:34.267362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afb3b5b-46ba-4a7b-b6af-152ac1a52795 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:34.276496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:34.277141 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.277420 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.277691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] PATH_INFO: `/v3/projects/7d25c483d30d4ac0b8818e5e6bdaaae3` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.278820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7d25c483d30d4ac0b8818e5e6bdaaae3)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:34.280943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:34.337273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7d25c483d30d4ac0b8818e5e6bdaaae3', 'request_id': 'req-7d149914-5bf2-42ee-a134-da0d0c042d61'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:34.343828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7d25c483d30d4ac0b8818e5e6bdaaae3', 'request_id': 'req-7d149914-5bf2-42ee-a134-da0d0c042d61'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:34.349531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7d25c483d30d4ac0b8818e5e6bdaaae3', 'request_id': 'req-7d149914-5bf2-42ee-a134-da0d0c042d61'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:34.367487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] The token cache is being invalidate because project 7d25c483d30d4ac0b8818e5e6bdaaae3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:34.367778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:34.368450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7d149914-5bf2-42ee-a134-da0d0c042d61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:34.369930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1100/2194] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:34 2022] DELETE /identity/v3/projects/7d25c483d30d4ac0b8818e5e6bdaaae3 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:34.374998 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c392a57d-f8ef-4623-9dbd-55c9328b09af None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.375282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c392a57d-f8ef-4623-9dbd-55c9328b09af None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.375553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c392a57d-f8ef-4623-9dbd-55c9328b09af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.402044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-c392a57d-f8ef-4623-9dbd-55c9328b09af None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:34.405437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c392a57d-f8ef-4623-9dbd-55c9328b09af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:34.485509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1095/2195] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:34.494792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc6561a-5527-4853-a3d1-fd8da5d8b4c4 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.495081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc6561a-5527-4853-a3d1-fd8da5d8b4c4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.495407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc6561a-5527-4853-a3d1-fd8da5d8b4c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.518233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4dc6561a-5527-4853-a3d1-fd8da5d8b4c4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:34.519898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4dc6561a-5527-4853-a3d1-fd8da5d8b4c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:34.529841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1101/2196] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:34.534071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c392a57d-f8ef-4623-9dbd-55c9328b09af None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:34.545925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c392a57d-f8ef-4623-9dbd-55c9328b09af None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:34.553010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:34.553970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.554069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.554366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.555077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:34.556930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:34.557958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:34.571916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1096/2197] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:34.576305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc6561a-5527-4853-a3d1-fd8da5d8b4c4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:34.595489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc6561a-5527-4853-a3d1-fd8da5d8b4c4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:34.607223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:34.607920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.608430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.608781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.609472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:34.611460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:34.652614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1102/2198] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:34 2022] POST /identity/v3/projects => generated 379 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:34.657010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:34.662754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:34.670250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7091045-9fa8-4d70-9171-2656617c264c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:34.736496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:34.737863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.738214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.738506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.739204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:34.741134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:34.764496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1097/2199] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:34 2022] POST /identity/v3/users => generated 290 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:34.769251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:34.780283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd4dade-531e-416f-a3fb-fb8b20f3a50e admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:34.785635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:34.786231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.786511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.786780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.787422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:34.789034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:34.802692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1103/2200] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:34.805056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:34.816679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bda62a-3214-4f71-824a-0dbf0fe2f059 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:34.822130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee1e207-b6f2-40ab-94e3-c280afb942f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:34.822714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee1e207-b6f2-40ab-94e3-c280afb942f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.822989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee1e207-b6f2-40ab-94e3-c280afb942f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.823256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee1e207-b6f2-40ab-94e3-c280afb942f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.824026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee1e207-b6f2-40ab-94e3-c280afb942f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:34.825743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee1e207-b6f2-40ab-94e3-c280afb942f9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:34.838145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1098/2201] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:34.842459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:34.853810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961f2187-2ff9-43c5-9d5c-2e023f0cf30d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:34.859484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:34.860162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.860444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.860717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] PATH_INFO: `/v3/projects/952f16556bfe4e208677b102f03fb5b4/users/320ac08c147947cca1d789eadc634ba0/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.876153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] RBAC: Authorizing `identity:create_grant(project_id=952f16556bfe4e208677b102f03fb5b4, user_id=320ac08c147947cca1d789eadc634ba0, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:34.879125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:34.893088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1104/2202] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:34 2022] PUT /identity/v3/projects/952f16556bfe4e208677b102f03fb5b4/users/320ac08c147947cca1d789eadc634ba0/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:34.904291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc378e48-5050-484e-a340-89acea96c8fc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:34.904665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc378e48-5050-484e-a340-89acea96c8fc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:34.905035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc378e48-5050-484e-a340-89acea96c8fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:34.936497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-dc378e48-5050-484e-a340-89acea96c8fc None None] MFA Rules not processed for user `320ac08c147947cca1d789eadc634ba0`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:34.937695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dc378e48-5050-484e-a340-89acea96c8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:35.007708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1099/2203] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:34 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:35.012280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.056112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:35.077935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29746471-4fc1-49d3-843c-a5b6eac5a601 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.083229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.084047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.084356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.084628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.085276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.087164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.114956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1105/2204] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:35 2022] POST /identity/v3/projects => generated 381 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:35.118863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc378e48-5050-484e-a340-89acea96c8fc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.130433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc378e48-5050-484e-a340-89acea96c8fc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.191669 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.192515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.192695 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.193039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.193728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.195742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.212386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:35.218708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1100/2205] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:35 2022] POST /identity/v3/users => generated 297 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:35.223198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.233637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12dc5b-884d-4235-a1f8-14ccd12d70e7 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.238463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.239048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.239326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.239595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.240270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.241848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.252575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1106/2206] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:35.257644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.267529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8959812-2665-43db-956e-346f589cff45 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.272494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.273146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.273420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.273688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] PATH_INFO: `/v3/projects/f3c744e720ef42a5a65b5136d7d37fe4/users/27cc46584dc4473b8bd995fc2e805a55/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.289225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3c744e720ef42a5a65b5136d7d37fe4, user_id=27cc46584dc4473b8bd995fc2e805a55, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.292421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.306595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1101/2207] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:35 2022] PUT /identity/v3/projects/f3c744e720ef42a5a65b5136d7d37fe4/users/27cc46584dc4473b8bd995fc2e805a55/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:35.310595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.321077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b61ffa9-2a78-4631-9b0c-3c11386e7ad4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.381129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2255d7-4e36-4ce6-a6a8-08c136d91c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.381735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2255d7-4e36-4ce6-a6a8-08c136d91c9a admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.382012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2255d7-4e36-4ce6-a6a8-08c136d91c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.382291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2255d7-4e36-4ce6-a6a8-08c136d91c9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.382914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2255d7-4e36-4ce6-a6a8-08c136d91c9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.384503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2255d7-4e36-4ce6-a6a8-08c136d91c9a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.395129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1107/2208] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:35 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:35.399422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.410385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92d259e-ac47-43a3-8912-a6cbd7abe042 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.415156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.415797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.416133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.416400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] PATH_INFO: `/v3/projects/f3c744e720ef42a5a65b5136d7d37fe4/users/27cc46584dc4473b8bd995fc2e805a55/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.418351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3c744e720ef42a5a65b5136d7d37fe4, user_id=27cc46584dc4473b8bd995fc2e805a55, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.421118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.432154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1102/2209] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:35 2022] PUT /identity/v3/projects/f3c744e720ef42a5a65b5136d7d37fe4/users/27cc46584dc4473b8bd995fc2e805a55/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:35.443512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b886b0af-182c-4b2d-8a95-166a3698b4a0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.443800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b886b0af-182c-4b2d-8a95-166a3698b4a0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.444105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b886b0af-182c-4b2d-8a95-166a3698b4a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.474819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b886b0af-182c-4b2d-8a95-166a3698b4a0 None None] MFA Rules not processed for user `27cc46584dc4473b8bd995fc2e805a55`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:35.476161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b886b0af-182c-4b2d-8a95-166a3698b4a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:35.551047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1108/2210] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:35 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:35.563878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.565764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:35.628564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb2db8e-7f23-43d2-b2a0-b805f8f3e874 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.633617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.634224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.634502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.634777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.645591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.647026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.657218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1103/2211] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:35 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:35.746150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b886b0af-182c-4b2d-8a95-166a3698b4a0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:35.757691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b886b0af-182c-4b2d-8a95-166a3698b4a0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:35.761670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e8157a-813d-4cda-b735-6cd911075276 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:35.764800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e8157a-813d-4cda-b735-6cd911075276 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:35.765097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e8157a-813d-4cda-b735-6cd911075276 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:35.765377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e8157a-813d-4cda-b735-6cd911075276 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:35.816157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:35.842739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e8157a-813d-4cda-b735-6cd911075276 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:35.844206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e8157a-813d-4cda-b735-6cd911075276 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:35.854406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1109/2212] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:35 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:45.652764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=41,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:45.850036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=12 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:48.914629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:48.978142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2c3569-6c99-47b1-945f-fdbbf9c2c465 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:48.982799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:48.983492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:48.983843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:48.984149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] PATH_INFO: `/v3/users/320ac08c147947cca1d789eadc634ba0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:48.985268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=320ac08c147947cca1d789eadc634ba0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:48.987910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:49.035809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '320ac08c147947cca1d789eadc634ba0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.036625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '320ac08c147947cca1d789eadc634ba0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.045056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '320ac08c147947cca1d789eadc634ba0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.045693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '320ac08c147947cca1d789eadc634ba0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.057777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '320ac08c147947cca1d789eadc634ba0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.069997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1104/2213] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:48 2022] DELETE /identity/v3/users/320ac08c147947cca1d789eadc634ba0 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:49.074464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:49.074921 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:49.075030 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:49.093924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:49.095180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:49.153474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1110/2214] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:49.163956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:49.165373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:49.222098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9566c3a6-63de-45e8-a95e-92fabbcf3ff1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:49.225811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:49.226407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:49.226683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:49.226949 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:49.236142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:49.238743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:49.249733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1105/2215] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:49.537207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:49.538929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:49.551750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e878a8c-9513-4e03-bc1f-9f17c299cdd3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:49.560848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:49.561509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:49.561934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:49.562213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] PATH_INFO: `/v3/projects/952f16556bfe4e208677b102f03fb5b4` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:49.563424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=952f16556bfe4e208677b102f03fb5b4)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:49.565724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:49.619162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '952f16556bfe4e208677b102f03fb5b4', 'request_id': 'req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.625399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '952f16556bfe4e208677b102f03fb5b4', 'request_id': 'req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.631867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '952f16556bfe4e208677b102f03fb5b4', 'request_id': 'req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.645704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] The token cache is being invalidate because project 952f16556bfe4e208677b102f03fb5b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:49.645989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.646597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.648125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1111/2216] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:49 2022] DELETE /identity/v3/projects/952f16556bfe4e208677b102f03fb5b4 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:49.652120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:49.662488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fba8dc-099b-4ab0-bbc4-483c78192a08 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:49.681450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:49.722935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:49.723685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:49.723963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:49.724236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] PATH_INFO: `/v3/users/27cc46584dc4473b8bd995fc2e805a55` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:49.725359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=27cc46584dc4473b8bd995fc2e805a55)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:49.727391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:49.777467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27cc46584dc4473b8bd995fc2e805a55'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.778319 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27cc46584dc4473b8bd995fc2e805a55'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.785958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27cc46584dc4473b8bd995fc2e805a55'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.786584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27cc46584dc4473b8bd995fc2e805a55'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.798580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-dbfb10cf-943c-42b5-bf8c-f406f763c5c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27cc46584dc4473b8bd995fc2e805a55'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:49.814520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1106/2217] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:49 2022] DELETE /identity/v3/users/27cc46584dc4473b8bd995fc2e805a55 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:50.130099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.131608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:50.145494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b646870d-0425-4f3b-8ca0-b5d96bae0ef7 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.208839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.209466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.209738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.210001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] PATH_INFO: `/v3/projects/f3c744e720ef42a5a65b5136d7d37fe4` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.211099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3c744e720ef42a5a65b5136d7d37fe4)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.213013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:50.242534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:50.260297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3c744e720ef42a5a65b5136d7d37fe4', 'request_id': 'req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:50.266710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3c744e720ef42a5a65b5136d7d37fe4', 'request_id': 'req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:50.272796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3c744e720ef42a5a65b5136d7d37fe4', 'request_id': 'req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:50.285102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] The token cache is being invalidate because project f3c744e720ef42a5a65b5136d7d37fe4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:50.285392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:50.286012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d0b19356-f3e2-4f80-8fc7-cc50917d28ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:50.287588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1112/2218] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:50 2022] DELETE /identity/v3/projects/f3c744e720ef42a5a65b5136d7d37fe4 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:50.292612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b0dd1b-8883-4171-9777-38b9b237ec7c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.293167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b0dd1b-8883-4171-9777-38b9b237ec7c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.293167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b0dd1b-8883-4171-9777-38b9b237ec7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.312885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-89b0dd1b-8883-4171-9777-38b9b237ec7c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:50.314060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-89b0dd1b-8883-4171-9777-38b9b237ec7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:50.373196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1107/2219] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:50.382169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd53cee7-0eee-4d2d-aa12-ed7908cb39b6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.382452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd53cee7-0eee-4d2d-aa12-ed7908cb39b6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.382721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd53cee7-0eee-4d2d-aa12-ed7908cb39b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.402356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-bd53cee7-0eee-4d2d-aa12-ed7908cb39b6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:50.403482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bd53cee7-0eee-4d2d-aa12-ed7908cb39b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:50.411588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1113/2220] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:50.416344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b0dd1b-8883-4171-9777-38b9b237ec7c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.426614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b0dd1b-8883-4171-9777-38b9b237ec7c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.431525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.432137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.432410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.432673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.433312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:50.433615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.435199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:50.447734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1108/2221] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:50.451928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd53cee7-0eee-4d2d-aa12-ed7908cb39b6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.461582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd53cee7-0eee-4d2d-aa12-ed7908cb39b6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.466299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.466893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.467172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.467440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.468118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.469992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:50.503770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1114/2222] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:50 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:50.508312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.520862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b79da6a-5291-4bac-be31-c0e8f794e308 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.551133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:50.589053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.589934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.590480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.591046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.591873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.594077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:50.617419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1109/2223] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:50 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:50.621591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.634782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c35a18-6ce0-4aa0-b6e6-48d181911189 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.640323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.641023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.641468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.641809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.642521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.644328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:50.660474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1115/2224] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:50.661634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.676461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab702568-a033-4ed8-837f-5b9c4e74d634 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.681948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271b8477-55d5-480d-a78b-927935b835d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.682553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b8477-55d5-480d-a78b-927935b835d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.682832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b8477-55d5-480d-a78b-927935b835d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.683167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271b8477-55d5-480d-a78b-927935b835d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.683984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271b8477-55d5-480d-a78b-927935b835d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.685589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271b8477-55d5-480d-a78b-927935b835d6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:50.697606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1110/2225] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:50.704165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.721847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c230a0-7aad-4137-a089-8318ff6b3dbe admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.729001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.729765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.730230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.730361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] PATH_INFO: `/v3/projects/9e45424117f44bcb8367055603579333/users/52f356458d224808966c7a10758f785c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.747829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e45424117f44bcb8367055603579333, user_id=52f356458d224808966c7a10758f785c, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.750780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:50.767493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1116/2226] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:50 2022] PUT /identity/v3/projects/9e45424117f44bcb8367055603579333/users/52f356458d224808966c7a10758f785c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:50.778597 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b5c8b-cd22-4352-81a1-c9152fd05133 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.778887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b5c8b-cd22-4352-81a1-c9152fd05133 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.779227 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b5c8b-cd22-4352-81a1-c9152fd05133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.830705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f14b5c8b-cd22-4352-81a1-c9152fd05133 None None] MFA Rules not processed for user `52f356458d224808966c7a10758f785c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:50.832124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f14b5c8b-cd22-4352-81a1-c9152fd05133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:50.911359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1111/2227] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:50.915739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:50.983205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b25325-8900-40b7-a031-9d9750df2b88 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:50.989137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:50.989763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:50.990122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:50.990407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:50.991125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:50.993492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.027138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1117/2228] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:50 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:51.028443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b5c8b-cd22-4352-81a1-c9152fd05133 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.040349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b5c8b-cd22-4352-81a1-c9152fd05133 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.105649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.106251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.106529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.106793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.107420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.109352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.136299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1112/2229] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:51 2022] POST /identity/v3/users => generated 299 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:51.137489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.148766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34538f11-542f-40ce-88db-cd6b7aa50273 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.154058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.154736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.155086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.155428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.156187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.157863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.169050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1118/2230] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:51.173159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.183921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8ea2d7-94e1-439f-9ce7-5e70a6711f5c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.188822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.189339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.189738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.190092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] PATH_INFO: `/v3/projects/33ee4152ae714594a54f6c4ff9d33e51/users/0638c72e88e2499a8fa7cd60f0c113b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.206551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33ee4152ae714594a54f6c4ff9d33e51, user_id=0638c72e88e2499a8fa7cd60f0c113b9, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.209297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.226069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1113/2231] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:51 2022] PUT /identity/v3/projects/33ee4152ae714594a54f6c4ff9d33e51/users/0638c72e88e2499a8fa7cd60f0c113b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:51.230162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.254864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58696870-e04b-4523-86e1-674d8f4d985c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.271893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:51.313435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59eb4f9-a517-425f-8539-7bf7e1a55da6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.314046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59eb4f9-a517-425f-8539-7bf7e1a55da6 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.314318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59eb4f9-a517-425f-8539-7bf7e1a55da6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.314631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59eb4f9-a517-425f-8539-7bf7e1a55da6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.315373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59eb4f9-a517-425f-8539-7bf7e1a55da6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.316976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59eb4f9-a517-425f-8539-7bf7e1a55da6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.327147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1119/2232] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:51 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:51.331819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.337460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:51.344821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888848a6-060b-401c-b779-86d33787bd57 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.350075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.350753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.351060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.351358 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] PATH_INFO: `/v3/projects/33ee4152ae714594a54f6c4ff9d33e51/users/0638c72e88e2499a8fa7cd60f0c113b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.353356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33ee4152ae714594a54f6c4ff9d33e51, user_id=0638c72e88e2499a8fa7cd60f0c113b9, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.356046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.367104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1114/2233] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:51 2022] PUT /identity/v3/projects/33ee4152ae714594a54f6c4ff9d33e51/users/0638c72e88e2499a8fa7cd60f0c113b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:51.380154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a260f19-5e00-416e-8ec0-f96cc270e02b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.380532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a260f19-5e00-416e-8ec0-f96cc270e02b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.380818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a260f19-5e00-416e-8ec0-f96cc270e02b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.408990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-2a260f19-5e00-416e-8ec0-f96cc270e02b None None] MFA Rules not processed for user `0638c72e88e2499a8fa7cd60f0c113b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:51.410442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2a260f19-5e00-416e-8ec0-f96cc270e02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:51.482321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1120/2234] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:51 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:51.494373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.496079 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:51.553339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf44750d-2c11-494f-9b98-9eaa70c4a938 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.557020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec1de39-5a30-480d-988f-91622134f5c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.557605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1de39-5a30-480d-988f-91622134f5c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.557878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1de39-5a30-480d-988f-91622134f5c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.558148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec1de39-5a30-480d-988f-91622134f5c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.567139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec1de39-5a30-480d-988f-91622134f5c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.568579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec1de39-5a30-480d-988f-91622134f5c9 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.577791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1115/2235] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:51 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:51.609455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a260f19-5e00-416e-8ec0-f96cc270e02b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.674100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a260f19-5e00-416e-8ec0-f96cc270e02b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.678712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.679352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.679628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.679896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] PATH_INFO: `/v3/users/52f356458d224808966c7a10758f785c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.681033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] RBAC: Authorizing `identity:delete_user(user_id=52f356458d224808966c7a10758f785c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.682940 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.729597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52f356458d224808966c7a10758f785c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.730185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52f356458d224808966c7a10758f785c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.730798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52f356458d224808966c7a10758f785c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.731998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:51.739455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52f356458d224808966c7a10758f785c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.751546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52f356458d224808966c7a10758f785c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.765038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1121/2236] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:51 2022] DELETE /identity/v3/users/52f356458d224808966c7a10758f785c => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:51.770204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.770638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.771003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.789857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:51.791032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:51.850390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1116/2237] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:51.860522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.862623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:51.901578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:51.902953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:51.916951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8cd9ee-31ea-4f9b-92d6-d31672be2475 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.921145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9e9a6b-9e41-4017-8db0-4f64c76729de admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:51.921462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.922068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.922342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.922606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] PATH_INFO: `/v3/users/efa27d88116c4f2595678a0ca800697e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.923706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] RBAC: Authorizing `identity:delete_user(user_id=efa27d88116c4f2595678a0ca800697e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.925316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e79ccf-a212-4313-bed6-ac8e8e626b0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:51.925909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e79ccf-a212-4313-bed6-ac8e8e626b0c service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:51.926188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e79ccf-a212-4313-bed6-ac8e8e626b0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:51.926439 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.926545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e79ccf-a212-4313-bed6-ac8e8e626b0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:51.935881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e79ccf-a212-4313-bed6-ac8e8e626b0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:51.937319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e79ccf-a212-4313-bed6-ac8e8e626b0c service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:51.948334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1122/2238] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:51.973832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'efa27d88116c4f2595678a0ca800697e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.974551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'efa27d88116c4f2595678a0ca800697e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.981527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'efa27d88116c4f2595678a0ca800697e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.982109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'efa27d88116c4f2595678a0ca800697e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:51.995442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'efa27d88116c4f2595678a0ca800697e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.009723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1117/2239] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:51 2022] DELETE /identity/v3/users/efa27d88116c4f2595678a0ca800697e => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:52.014329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.014654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.014925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.034908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:52.036128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:52.102671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1123/2240] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:52.109909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:52.111208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:52.118922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:52.175838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516c5ba4-54c4-4d51-9dc2-d8b0a996b84f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:52.180225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:52.180833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.181106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.181369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.191531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:52.193007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:52.203003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1118/2241] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:52.236182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:52.238073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:52.253135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff67c35-bd75-4ab4-99a0-3aaf557a7259 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:52.261413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:52.262459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.262848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.263236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] PATH_INFO: `/v3/projects/9e45424117f44bcb8367055603579333` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.264578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e45424117f44bcb8367055603579333)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:52.267492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:52.328175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e45424117f44bcb8367055603579333', 'request_id': 'req-feb56b0f-801e-4703-9569-d1fc72e5a7ec'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.339586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e45424117f44bcb8367055603579333', 'request_id': 'req-feb56b0f-801e-4703-9569-d1fc72e5a7ec'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.347354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e45424117f44bcb8367055603579333', 'request_id': 'req-feb56b0f-801e-4703-9569-d1fc72e5a7ec'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.350202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:52.371124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] The token cache is being invalidate because project 9e45424117f44bcb8367055603579333 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:52.371325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.372509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.374419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1124/2242] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:52 2022] DELETE /identity/v3/projects/9e45424117f44bcb8367055603579333 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:52.378802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:52.388942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb21e518-e565-45f7-9144-af5df387dee2 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:52.489430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:52.490198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.490486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.490762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] PATH_INFO: `/v3/users/0638c72e88e2499a8fa7cd60f0c113b9` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.491929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=0638c72e88e2499a8fa7cd60f0c113b9)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:52.494174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:52.549442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0638c72e88e2499a8fa7cd60f0c113b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.550588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0638c72e88e2499a8fa7cd60f0c113b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.558368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0638c72e88e2499a8fa7cd60f0c113b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.558963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0638c72e88e2499a8fa7cd60f0c113b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.571542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:52.573036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:52.574901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-c379bd31-a450-48ff-9654-9d63c3360e3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0638c72e88e2499a8fa7cd60f0c113b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.589295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb56b0f-801e-4703-9569-d1fc72e5a7ec admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:52.594510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1119/2243] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:52 2022] DELETE /identity/v3/users/0638c72e88e2499a8fa7cd60f0c113b9 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:52.680709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:52.681344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.681625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.681891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] PATH_INFO: `/v3/projects/606596b4dfad42ecae365c057b989f19` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.682996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=606596b4dfad42ecae365c057b989f19)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:52.684986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:52.734848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '606596b4dfad42ecae365c057b989f19', 'request_id': 'req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.742384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '606596b4dfad42ecae365c057b989f19', 'request_id': 'req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.749107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '606596b4dfad42ecae365c057b989f19', 'request_id': 'req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.774126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] The token cache is being invalidate because project 606596b4dfad42ecae365c057b989f19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:52.774422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.775623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d9b311eb-d81c-429c-9ad3-36e7e40fa4f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.778594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1125/2244] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:52 2022] DELETE /identity/v3/projects/606596b4dfad42ecae365c057b989f19 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:52.783881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.784201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.784522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.813867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:52.815582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:52.836742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:52.881730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1120/2245] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:52.891361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0f0c0-9295-4967-861f-5a70f9e0cfd4 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.891726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0f0c0-9295-4967-861f-5a70f9e0cfd4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.892145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0f0c0-9295-4967-861f-5a70f9e0cfd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.901400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:52.903160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:52.916513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b8f0f0c0-9295-4967-861f-5a70f9e0cfd4 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:52.917728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b8f0f0c0-9295-4967-861f-5a70f9e0cfd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:52.918374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a89749-b070-4c81-8c05-8fc83732e956 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:52.923324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:52.924616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.925041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.925377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] PATH_INFO: `/v3/projects/33ee4152ae714594a54f6c4ff9d33e51` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.926499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33ee4152ae714594a54f6c4ff9d33e51)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:52.927786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1126/2246] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:52.928799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:52.932109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0f0c0-9295-4967-861f-5a70f9e0cfd4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:52.942959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0f0c0-9295-4967-861f-5a70f9e0cfd4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:52.948101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:52.948697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:52.948974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:52.949242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:52.949884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:52.950182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:52.951758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:52.964642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1127/2247] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:52.968710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:52.979223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda44993-7ccf-4941-8946-3ff0f9eb997c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:52.983331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33ee4152ae714594a54f6c4ff9d33e51', 'request_id': 'req-43247169-4d2d-4f78-84c1-15cdbe00d604'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.990565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33ee4152ae714594a54f6c4ff9d33e51', 'request_id': 'req-43247169-4d2d-4f78-84c1-15cdbe00d604'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:52.996323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33ee4152ae714594a54f6c4ff9d33e51', 'request_id': 'req-43247169-4d2d-4f78-84c1-15cdbe00d604'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:53.010412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] The token cache is being invalidate because project 33ee4152ae714594a54f6c4ff9d33e51 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:53.010678 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:53.011317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-43247169-4d2d-4f78-84c1-15cdbe00d604 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:53.013008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1121/2248] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:52 2022] DELETE /identity/v3/projects/33ee4152ae714594a54f6c4ff9d33e51 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:53.017430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1a422-a55a-437d-acb5-22877f13a678 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.017743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1a422-a55a-437d-acb5-22877f13a678 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.018041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e1a422-a55a-437d-acb5-22877f13a678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.038132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b0e1a422-a55a-437d-acb5-22877f13a678 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:53.039337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b0e1a422-a55a-437d-acb5-22877f13a678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:53.039644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.040285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.040673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.040954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.041603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.043483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.047286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1122/2249] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:53.049789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=2,keystone:DELETE=8,keystone:SELECT=90 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:53.056787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28319bbe-b4f0-42be-aca0-2813a64643ae None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.057178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28319bbe-b4f0-42be-aca0-2813a64643ae None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.057491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28319bbe-b4f0-42be-aca0-2813a64643ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.079154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1128/2250] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:52 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:53.079381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-28319bbe-b4f0-42be-aca0-2813a64643ae None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:53.079626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-28319bbe-b4f0-42be-aca0-2813a64643ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:53.080150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.081703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:53.140864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1123/2251] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:53.145245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28319bbe-b4f0-42be-aca0-2813a64643ae None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.146694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce66cf6-475f-4d72-accf-d0100b0146b5 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.151388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.152015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.152301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.152570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.153197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.154121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28319bbe-b4f0-42be-aca0-2813a64643ae None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.155121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.159567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.160525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.160965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.161236 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.161936 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:53.162234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.164065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.181224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1124/2252] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:53.181401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1129/2253] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/users => generated 289 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:53.182065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.182565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.191964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5baa0429-0b97-4e16-9b41-b7854283e6b6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.193064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356b3c10-3cf3-4a75-96a7-019cf1425156 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.196565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.197145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.197424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.197691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.197804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.198276 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.198390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.198553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.198847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.199471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.199884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.201385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.210345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1125/2254] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:53.214214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.224852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb575e2-9df6-4379-9c0d-0b40b99e7943 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.229474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.230111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.230429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.230757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.231263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1130/2255] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/projects => generated 379 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:53.232148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.233848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.235389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.248497 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1126/2256] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:53.250325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.261748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e833fe-48cc-4f05-846d-2506d2f36920 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.266684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:53.304626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a044252f-0f5f-4281-a997-44b3b0c2d5d5 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.316442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.317063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.317337 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.317604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.318232 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.320368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.332797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77e621f-d95f-47fe-8d72-83648168e0ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.333800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77e621f-d95f-47fe-8d72-83648168e0ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.334145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77e621f-d95f-47fe-8d72-83648168e0ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.334455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77e621f-d95f-47fe-8d72-83648168e0ad admin admin] PATH_INFO: `/v3/projects/a93da81805a145908bddc4c70d77c8ca/users/26fb83174b96449b85a381fcc0e9fc2b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.342076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1131/2257] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/users => generated 290 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:53.347206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.350751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77e621f-d95f-47fe-8d72-83648168e0ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=a93da81805a145908bddc4c70d77c8ca, user_id=26fb83174b96449b85a381fcc0e9fc2b, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.353496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77e621f-d95f-47fe-8d72-83648168e0ad admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.356720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5b2565-7b96-4b75-8fb2-3ac6e6174eb8 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.361726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.362573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.362687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.362871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.363507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.365097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.366628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1127/2258] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:53 2022] PUT /identity/v3/projects/a93da81805a145908bddc4c70d77c8ca/users/26fb83174b96449b85a381fcc0e9fc2b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:53.376537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1132/2259] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:53.378084 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.378372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.378642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.379104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.391758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741faaaa-d150-438d-b889-c4af272592d8 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.410141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] MFA Rules not processed for user `26fb83174b96449b85a381fcc0e9fc2b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:53.411357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:53.425691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:53.456196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.456854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.457082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.457369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.457995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.459579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.474045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1133/2260] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:53.475052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.479926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1128/2261] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:53.494426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.495820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:53.495962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc7a535-57aa-47d5-93e2-3e26a8ed811c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.500750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80751185-cd35-4486-ae3d-5a29e6ce171d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.501341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80751185-cd35-4486-ae3d-5a29e6ce171d admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.501749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80751185-cd35-4486-ae3d-5a29e6ce171d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.502085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80751185-cd35-4486-ae3d-5a29e6ce171d admin admin] PATH_INFO: `/v3/projects/587b968560f24c82a1572ed761c9bb37/users/ece0b343670b48e8984bf7d9d33ac584/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.522369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80751185-cd35-4486-ae3d-5a29e6ce171d admin admin] RBAC: Authorizing `identity:create_grant(project_id=587b968560f24c82a1572ed761c9bb37, user_id=ece0b343670b48e8984bf7d9d33ac584, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.525703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80751185-cd35-4486-ae3d-5a29e6ce171d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.542754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1134/2262] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:53 2022] PUT /identity/v3/projects/587b968560f24c82a1572ed761c9bb37/users/ece0b343670b48e8984bf7d9d33ac584/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:53.554756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0699d2c1-4ba3-4311-b91a-62f7a03529c0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.555256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0699d2c1-4ba3-4311-b91a-62f7a03529c0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.555618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0699d2c1-4ba3-4311-b91a-62f7a03529c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.565907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7fd8e60-0f97-45c1-b475-85c6de58513f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.589827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0699d2c1-4ba3-4311-b91a-62f7a03529c0 None None] MFA Rules not processed for user `ece0b343670b48e8984bf7d9d33ac584`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:53.591057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0699d2c1-4ba3-4311-b91a-62f7a03529c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:53.604114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:53.625506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.626122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.626395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.626657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.684536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1135/2263] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:53.689102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0699d2c1-4ba3-4311-b91a-62f7a03529c0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.708067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.709763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.720485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1129/2264] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:53 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:55:53.762377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0699d2c1-4ba3-4311-b91a-62f7a03529c0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.768166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.768545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.768836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.769108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.770252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.772320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.817166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1136/2265] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/projects => generated 381 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:53.821339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.842388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0242406-3ae9-4014-8276-d1b2537f920f service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.916527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.917135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.917792 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.918066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.919159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.921990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.958698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:53.959860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1130/2266] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:53 2022] POST /identity/v3/users => generated 297 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:53.970355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a86f45-deb6-49b9-9281-fa8d1d2a52cb admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:53.977351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:53.978093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:53.978483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:53.978809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:53.979459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:53.981547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:53.994636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1137/2267] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:55:53.998920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.010380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e48909-4d75-4b9e-bd8a-8971bfa08296 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.020646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.022262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.022561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.022844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] PATH_INFO: `/v3/projects/306fc760ad75439e8c6d56b3bc1c2a49/users/b1199c0638e84f2c8f035fdb6b282ebe/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.041411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=306fc760ad75439e8c6d56b3bc1c2a49, user_id=b1199c0638e84f2c8f035fdb6b282ebe, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:54.044106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:54.060310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1131/2268] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:53 2022] PUT /identity/v3/projects/306fc760ad75439e8c6d56b3bc1c2a49/users/b1199c0638e84f2c8f035fdb6b282ebe/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:54.064250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.076366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b85b87-99c1-4f13-beda-b976b0061a7d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.091011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:54.146636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cc2c77-b606-4770-9404-d41e23b85bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.147439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc2c77-b606-4770-9404-d41e23b85bc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.147688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc2c77-b606-4770-9404-d41e23b85bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.147980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cc2c77-b606-4770-9404-d41e23b85bc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.148778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cc2c77-b606-4770-9404-d41e23b85bc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:54.150509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cc2c77-b606-4770-9404-d41e23b85bc3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:54.161213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1138/2269] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:54 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:54.165476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.176703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca8dc91-21be-4359-ba68-91b1f34a5b74 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.181339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.181964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.182240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.182508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] PATH_INFO: `/v3/projects/306fc760ad75439e8c6d56b3bc1c2a49/users/b1199c0638e84f2c8f035fdb6b282ebe/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.184446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] RBAC: Authorizing `identity:create_grant(project_id=306fc760ad75439e8c6d56b3bc1c2a49, user_id=b1199c0638e84f2c8f035fdb6b282ebe, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:54.187073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:54.200276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1132/2270] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:54 2022] PUT /identity/v3/projects/306fc760ad75439e8c6d56b3bc1c2a49/users/b1199c0638e84f2c8f035fdb6b282ebe/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:54.211352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c83c54-b85e-48f2-b88c-cfc9e2b2cdc9 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.211708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c83c54-b85e-48f2-b88c-cfc9e2b2cdc9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.212095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c83c54-b85e-48f2-b88c-cfc9e2b2cdc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.241611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-54c83c54-b85e-48f2-b88c-cfc9e2b2cdc9 None None] MFA Rules not processed for user `b1199c0638e84f2c8f035fdb6b282ebe`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:54.242915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-54c83c54-b85e-48f2-b88c-cfc9e2b2cdc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:54.317117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1139/2271] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:54 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:55:54.329705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.331888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:54.382070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:54.401268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c4aa2-debf-458c-bd11-51ff4421d5af admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.405369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.406008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.406286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.406551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.419350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:54.421026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:54.433086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1133/2272] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:54 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:54.766176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c83c54-b85e-48f2-b88c-cfc9e2b2cdc9 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.776043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c83c54-b85e-48f2-b88c-cfc9e2b2cdc9 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.780570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b1199c0638e84f2c8f035fdb6b282ebe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '306fc760ad75439e8c6d56b3bc1c2a49', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.781158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.781441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.781712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.782355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:54.782651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:54.784935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:54.796327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1140/2273] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:54.800604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.809838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6991b-75b2-4de6-a250-57ae0a272b68 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.814481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b1199c0638e84f2c8f035fdb6b282ebe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '306fc760ad75439e8c6d56b3bc1c2a49', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.815082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.815356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.815628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.816294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:54.818192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:54.844764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1134/2274] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:54 2022] POST /identity/v3/projects => generated 354 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:54.850558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.861325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4065dd59-2cfb-4c72-818d-16a798a049e3 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.923839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b1199c0638e84f2c8f035fdb6b282ebe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '306fc760ad75439e8c6d56b3bc1c2a49', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.924642 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.924995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.925278 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.925960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:54.926259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:54.927902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:54.940199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1141/2275] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:54.982765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:54.993159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fb030f-8b3f-4a60-9529-249087835631 tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:54.997687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b1199c0638e84f2c8f035fdb6b282ebe', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '306fc760ad75439e8c6d56b3bc1c2a49', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:54.998296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:54.998576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:54.998843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] PATH_INFO: `/v3/projects/31847dccf35942e9b319871a31a85941` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:54.999951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorizing `identity:delete_project(project_id=31847dccf35942e9b319871a31a85941)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:55.001938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:55.049650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31847dccf35942e9b319871a31a85941', 'request_id': 'req-61062420-41e2-4e66-b5f4-d73c1a70a00b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.056235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31847dccf35942e9b319871a31a85941', 'request_id': 'req-61062420-41e2-4e66-b5f4-d73c1a70a00b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.061642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31847dccf35942e9b319871a31a85941', 'request_id': 'req-61062420-41e2-4e66-b5f4-d73c1a70a00b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.073643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] The token cache is being invalidate because project 31847dccf35942e9b319871a31a85941 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:55.073923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.074543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-61062420-41e2-4e66-b5f4-d73c1a70a00b tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.076029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1135/2276] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:54 2022] DELETE /identity/v3/projects/31847dccf35942e9b319871a31a85941 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:55.295269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:55.297027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:55.312549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:55.368744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa1319d-0836-433a-b7a7-ae0e928309aa tempest-VolumeQuotaClassesTest-1156000297 tempest-VolumeQuotaClassesTest-1156000297-project-admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:55.425542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:55.426222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:55.426498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:55.426764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] PATH_INFO: `/v3/users/ece0b343670b48e8984bf7d9d33ac584` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:55.427887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ece0b343670b48e8984bf7d9d33ac584)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:55.429820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:55.474578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ece0b343670b48e8984bf7d9d33ac584'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.475274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ece0b343670b48e8984bf7d9d33ac584'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.475887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ece0b343670b48e8984bf7d9d33ac584'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.483318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ece0b343670b48e8984bf7d9d33ac584'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.496772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ece0b343670b48e8984bf7d9d33ac584'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:55.510271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1142/2277] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:55 2022] DELETE /identity/v3/users/ece0b343670b48e8984bf7d9d33ac584 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:55.514700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:55.515219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:55.515219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:55.535441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:55.536688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:55.598259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1136/2278] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:55.608164 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:55.609673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:55.654772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:55.672486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8bafc-f4e1-4adf-b4a1-63913c61c476 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:55.676285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:55.676889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:55.677170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:55.677437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:55.690958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:55.692423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:55.701985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1143/2279] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:55.968959 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:55.970368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:55.982391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5868a8b7-7252-4c8c-a705-eb52aca61c34 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:55.987376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:55.988058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:55.988294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:55.988640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] PATH_INFO: `/v3/projects/587b968560f24c82a1572ed761c9bb37` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:55.989810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] RBAC: Authorizing `identity:delete_project(project_id=587b968560f24c82a1572ed761c9bb37)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:55.991743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:56.038407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:56.043230 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '587b968560f24c82a1572ed761c9bb37', 'request_id': 'req-49336625-5586-4589-b56d-b043ca382931'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.051876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '587b968560f24c82a1572ed761c9bb37', 'request_id': 'req-49336625-5586-4589-b56d-b043ca382931'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.057554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '587b968560f24c82a1572ed761c9bb37', 'request_id': 'req-49336625-5586-4589-b56d-b043ca382931'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.070617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] The token cache is being invalidate because project 587b968560f24c82a1572ed761c9bb37 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:56.070906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.071526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.073218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1137/2280] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:55 2022] DELETE /identity/v3/projects/587b968560f24c82a1572ed761c9bb37 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:56.077199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:56.087526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7c45bb-c863-499c-a6ca-43cdabfa61ee service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:56.151457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:56.152038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:56.152323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:56.152597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] PATH_INFO: `/v3/users/b1199c0638e84f2c8f035fdb6b282ebe` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:56.153721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1199c0638e84f2c8f035fdb6b282ebe)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:56.155638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:56.203915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1199c0638e84f2c8f035fdb6b282ebe'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.204665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1199c0638e84f2c8f035fdb6b282ebe'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.205331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1199c0638e84f2c8f035fdb6b282ebe'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.212187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1199c0638e84f2c8f035fdb6b282ebe'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.224555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-07b4c6e2-2d6a-4c27-8642-80128b873dfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1199c0638e84f2c8f035fdb6b282ebe'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.237651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1144/2281] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:56 2022] DELETE /identity/v3/users/b1199c0638e84f2c8f035fdb6b282ebe => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:56.512323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:56.513754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:56.527733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49336625-5586-4589-b56d-b043ca382931 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:56.585843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:56.586467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:56.586744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:56.587012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] PATH_INFO: `/v3/projects/306fc760ad75439e8c6d56b3bc1c2a49` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:56.588173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] RBAC: Authorizing `identity:delete_project(project_id=306fc760ad75439e8c6d56b3bc1c2a49)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:56.590071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:56.637479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '306fc760ad75439e8c6d56b3bc1c2a49', 'request_id': 'req-761ef975-8ccf-42fc-bbbc-69e01501938e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.644600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '306fc760ad75439e8c6d56b3bc1c2a49', 'request_id': 'req-761ef975-8ccf-42fc-bbbc-69e01501938e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.650520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '306fc760ad75439e8c6d56b3bc1c2a49', 'request_id': 'req-761ef975-8ccf-42fc-bbbc-69e01501938e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.663327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] The token cache is being invalidate because project 306fc760ad75439e8c6d56b3bc1c2a49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:56.663608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.664244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-761ef975-8ccf-42fc-bbbc-69e01501938e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:56.665706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1138/2282] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:56 2022] DELETE /identity/v3/projects/306fc760ad75439e8c6d56b3bc1c2a49 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:56.670622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386193b7-e5f9-4041-9d5c-b1d03d881e60 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:56.671098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386193b7-e5f9-4041-9d5c-b1d03d881e60 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:56.671199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-386193b7-e5f9-4041-9d5c-b1d03d881e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:56.692250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-386193b7-e5f9-4041-9d5c-b1d03d881e60 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:56.693361 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-386193b7-e5f9-4041-9d5c-b1d03d881e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:56.747730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:56.765601 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1145/2283] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:56.767301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39958a83-ac78-4137-9f4a-ddcb98c1cd69 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:56.768263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39958a83-ac78-4137-9f4a-ddcb98c1cd69 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:56.768558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39958a83-ac78-4137-9f4a-ddcb98c1cd69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:56.789446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-39958a83-ac78-4137-9f4a-ddcb98c1cd69 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:56.790851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-39958a83-ac78-4137-9f4a-ddcb98c1cd69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:56.802509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1139/2284] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:56.807069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386193b7-e5f9-4041-9d5c-b1d03d881e60 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:56.818103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-386193b7-e5f9-4041-9d5c-b1d03d881e60 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:56.822823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:56.823428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:56.823718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:56.824021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:56.824671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:56.825002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:56.826621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:56.839976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1146/2285] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:55:56.845014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39958a83-ac78-4137-9f4a-ddcb98c1cd69 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:56.854643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39958a83-ac78-4137-9f4a-ddcb98c1cd69 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:56.859370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:56.859980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:56.860257 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:56.860521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:56.861173 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:56.863210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:56.891098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1140/2286] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:56 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:56.895916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:56.905898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d82e91f-9235-4596-a7bb-b95e2ed27d76 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:56.962799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:56.963693 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:56.964239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:56.964719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:56.965911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:56.969795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:56.992513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1147/2287] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:56 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:55:56.996628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:57.007842 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f948181-181f-4a73-93cc-ee806a3983d1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:57.012454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:57.013037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.013310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.013579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.014204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:57.015777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:57.026083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1141/2288] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:57.029958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:57.040089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322b357b-4a40-40e6-b038-dae52c816c0c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:57.044652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b69802-c2c8-4734-98c2-bed0f97b7d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:57.045231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b69802-c2c8-4734-98c2-bed0f97b7d63 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.045528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b69802-c2c8-4734-98c2-bed0f97b7d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.045778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b69802-c2c8-4734-98c2-bed0f97b7d63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.046440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b69802-c2c8-4734-98c2-bed0f97b7d63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:57.048224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b69802-c2c8-4734-98c2-bed0f97b7d63 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:57.060904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1148/2289] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:57.065631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:57.075677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4963f1f5-f7e1-405b-bd71-f1121012c48a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:57.080432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:57.081064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.081341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.081609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] PATH_INFO: `/v3/projects/95f82ffbfe2c471ba841cf6bd4e3ce44/users/08e03c2c1d4c4ad88608efe1281fe3ce/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.096023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=95f82ffbfe2c471ba841cf6bd4e3ce44, user_id=08e03c2c1d4c4ad88608efe1281fe3ce, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:57.098790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:57.106742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:57.113730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1142/2290] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:57 2022] PUT /identity/v3/projects/95f82ffbfe2c471ba841cf6bd4e3ce44/users/08e03c2c1d4c4ad88608efe1281fe3ce/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:57.124952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbf0de-026d-453e-ae41-d0a6040f6108 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.125235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbf0de-026d-453e-ae41-d0a6040f6108 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.125529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbf0de-026d-453e-ae41-d0a6040f6108 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.153584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-cdbbf0de-026d-453e-ae41-d0a6040f6108 None None] MFA Rules not processed for user `08e03c2c1d4c4ad88608efe1281fe3ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:57.154817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cdbbf0de-026d-453e-ae41-d0a6040f6108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:57.219178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1149/2291] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:57 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:57.230896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:57.232568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:57.292250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58e2395-37aa-4afd-9b0f-8bed323116cd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:57.296325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1421bc-439d-4901-8541-4dc23dc7fb3a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:57.296923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1421bc-439d-4901-8541-4dc23dc7fb3a service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.297197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1421bc-439d-4901-8541-4dc23dc7fb3a service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.297459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1421bc-439d-4901-8541-4dc23dc7fb3a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.306730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1421bc-439d-4901-8541-4dc23dc7fb3a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:57.308180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1421bc-439d-4901-8541-4dc23dc7fb3a service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:57.317789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1143/2292] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:57 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:57.339032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbf0de-026d-453e-ae41-d0a6040f6108 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:57.407125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:57.414863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbf0de-026d-453e-ae41-d0a6040f6108 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:57.419718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:57.420383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.420662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.421007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] PATH_INFO: `/v3/users/08e03c2c1d4c4ad88608efe1281fe3ce` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.422160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] RBAC: Authorizing `identity:delete_user(user_id=08e03c2c1d4c4ad88608efe1281fe3ce)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:57.424820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:57.470205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08e03c2c1d4c4ad88608efe1281fe3ce'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:57.471261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08e03c2c1d4c4ad88608efe1281fe3ce'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:57.471926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08e03c2c1d4c4ad88608efe1281fe3ce'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:57.478674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08e03c2c1d4c4ad88608efe1281fe3ce'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:57.491779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08e03c2c1d4c4ad88608efe1281fe3ce'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:57.503803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1150/2293] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:55:57 2022] DELETE /identity/v3/users/08e03c2c1d4c4ad88608efe1281fe3ce => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:55:57.508529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.508847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.509125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.527513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:57.528731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:57.587344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1144/2294] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:57.597306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:57.598726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:57.656051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f1624c-4d9a-46cd-ac03-fe045072111b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:57.660396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b5a5cc-a5e6-4c82-aea9-87b7ece1f2ce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:57.660985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b5a5cc-a5e6-4c82-aea9-87b7ece1f2ce service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:57.661538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b5a5cc-a5e6-4c82-aea9-87b7ece1f2ce service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:57.661835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b5a5cc-a5e6-4c82-aea9-87b7ece1f2ce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:57.671008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b5a5cc-a5e6-4c82-aea9-87b7ece1f2ce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:57.672455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b5a5cc-a5e6-4c82-aea9-87b7ece1f2ce service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:57.682149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1151/2295] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:55:57.992602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:57.994029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:58.006563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad061102-3db4-4660-b771-05d40d9831f7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.016239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.016908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.017180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.017474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] PATH_INFO: `/v3/projects/95f82ffbfe2c471ba841cf6bd4e3ce44` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.019062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95f82ffbfe2c471ba841cf6bd4e3ce44)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.021022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.081742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95f82ffbfe2c471ba841cf6bd4e3ce44', 'request_id': 'req-19732d00-f4a4-44cc-872c-58b70d4668a9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:58.088063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95f82ffbfe2c471ba841cf6bd4e3ce44', 'request_id': 'req-19732d00-f4a4-44cc-872c-58b70d4668a9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:58.093466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95f82ffbfe2c471ba841cf6bd4e3ce44', 'request_id': 'req-19732d00-f4a4-44cc-872c-58b70d4668a9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:58.105609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] The token cache is being invalidate because project 95f82ffbfe2c471ba841cf6bd4e3ce44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:55:58.105896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:58.106518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-19732d00-f4a4-44cc-872c-58b70d4668a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:55:58.108026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1145/2296] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:55:57 2022] DELETE /identity/v3/projects/95f82ffbfe2c471ba841cf6bd4e3ce44 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:58.113774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407cba63-9ef8-4bd4-9ac5-f16180c86dd5 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.114060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407cba63-9ef8-4bd4-9ac5-f16180c86dd5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.114330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407cba63-9ef8-4bd4-9ac5-f16180c86dd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.135984 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-407cba63-9ef8-4bd4-9ac5-f16180c86dd5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:58.137174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-407cba63-9ef8-4bd4-9ac5-f16180c86dd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:58.196834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1152/2297] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:58.204481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298a439-1138-4052-bdb4-44f8811bd4b7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.204764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298a439-1138-4052-bdb4-44f8811bd4b7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.205230 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d298a439-1138-4052-bdb4-44f8811bd4b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.225830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d298a439-1138-4052-bdb4-44f8811bd4b7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:58.227009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d298a439-1138-4052-bdb4-44f8811bd4b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:58.234900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1146/2298] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:58.239613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407cba63-9ef8-4bd4-9ac5-f16180c86dd5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:58.249235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407cba63-9ef8-4bd4-9ac5-f16180c86dd5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.253817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.254661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.254939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.255207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.255842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:55:58.256176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.257742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.269216 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1153/2299] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:55:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:55:58.273414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298a439-1138-4052-bdb4-44f8811bd4b7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:58.282771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d298a439-1138-4052-bdb4-44f8811bd4b7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.287985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.288565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.288839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.289105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.289719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.291863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.309005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:58.322499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1147/2300] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:55:58 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:58.326609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:58.333296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:55:58.337444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d629e1-5b40-4134-9d64-7205c109fd16 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.396362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.396974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.397257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.397541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.398160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.400186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.429496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1154/2301] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:55:58 2022] POST /identity/v3/users => generated 293 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:55:58.430726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:58.441514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd6f213-56e2-44af-93f0-73eea4322c61 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.446252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.446841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.447118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.447386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.448049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.449812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.460640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1148/2302] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:58.464906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:58.476517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13061d77-8366-44d5-ad2b-67220adb8e50 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.481315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf56d793-7082-47b7-af86-90cdc7fb16a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.482039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf56d793-7082-47b7-af86-90cdc7fb16a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.482317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf56d793-7082-47b7-af86-90cdc7fb16a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.482582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf56d793-7082-47b7-af86-90cdc7fb16a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.483224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf56d793-7082-47b7-af86-90cdc7fb16a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.484894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf56d793-7082-47b7-af86-90cdc7fb16a9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.497279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1155/2303] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:55:58.501294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:58.512214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a8ebb5-05ec-4072-aa62-fb5dd2c8c259 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.516946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.518140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.518415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.518682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] PATH_INFO: `/v3/projects/91aabdd330f74058b7a980f075092142/users/cd2d28e771c94fde85e81a2a8d882cba/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.534733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] RBAC: Authorizing `identity:create_grant(project_id=91aabdd330f74058b7a980f075092142, user_id=cd2d28e771c94fde85e81a2a8d882cba, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.537384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.551708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1149/2304] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:55:58 2022] PUT /identity/v3/projects/91aabdd330f74058b7a980f075092142/users/cd2d28e771c94fde85e81a2a8d882cba/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:55:58.562120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae1da1d-2e6a-4ff0-a81b-06903ecfc8b3 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.562407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae1da1d-2e6a-4ff0-a81b-06903ecfc8b3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.562673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae1da1d-2e6a-4ff0-a81b-06903ecfc8b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.590939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-fae1da1d-2e6a-4ff0-a81b-06903ecfc8b3 None None] MFA Rules not processed for user `cd2d28e771c94fde85e81a2a8d882cba`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:55:58.592217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fae1da1d-2e6a-4ff0-a81b-06903ecfc8b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:58.656734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1156/2305] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:55:58 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:55:58.668216 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:55:58.669891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:55:58.726515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b997006e-4890-4f8d-9577-a439ac39a598 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:55:58.730199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:55:58.730791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:55:58.731069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:55:58.731354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:55:58.739650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:55:58.741114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:55:58.749818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1150/2306] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:55:58 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:08.653804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=64,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:08.745793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=43,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:13.968709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d9500e-c474-48fd-95cd-0aed081a05fa None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:13.969209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d9500e-c474-48fd-95cd-0aed081a05fa None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:13.969335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d9500e-c474-48fd-95cd-0aed081a05fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:13.989147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c8d9500e-c474-48fd-95cd-0aed081a05fa None None] MFA Rules not processed for user `47a422989ce94168a042135234bea46b`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:13.990324 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c8d9500e-c474-48fd-95cd-0aed081a05fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:14.044441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1157/2307] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:13 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:14.474985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:14.544426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e726ab10-82c2-438b-a845-09add58f35b0 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:14.549781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:14.550416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:14.550748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:14.551120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] PATH_INFO: `/v3/users/cd2d28e771c94fde85e81a2a8d882cba` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:14.552504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd2d28e771c94fde85e81a2a8d882cba)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:14.554564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:14.611568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd2d28e771c94fde85e81a2a8d882cba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:14.612318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd2d28e771c94fde85e81a2a8d882cba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:14.620559 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd2d28e771c94fde85e81a2a8d882cba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:14.621314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd2d28e771c94fde85e81a2a8d882cba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:14.636895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd2d28e771c94fde85e81a2a8d882cba'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:14.654627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1151/2308] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:56:14 2022] DELETE /identity/v3/users/cd2d28e771c94fde85e81a2a8d882cba => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:14.661150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:14.661843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:14.662341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:14.701638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:14.702878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:14.768670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1158/2309] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:14.778769 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:14.781121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:14.866380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed38bc75-a573-4537-ac24-89d961149b7c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:14.871176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9a9cd8-21fd-4180-bc97-61348ffe2f2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:14.872033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a9cd8-21fd-4180-bc97-61348ffe2f2b service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:14.872413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a9cd8-21fd-4180-bc97-61348ffe2f2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:14.872770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9a9cd8-21fd-4180-bc97-61348ffe2f2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:14.887568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9a9cd8-21fd-4180-bc97-61348ffe2f2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:14.889036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9a9cd8-21fd-4180-bc97-61348ffe2f2b service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:14.913857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1152/2310] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:56:15.398537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:15.400126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:15.420271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798fe00-8c85-499b-80b4-452147e4cc13 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:15.426802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:15.427430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.427713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.428002 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] PATH_INFO: `/v3/projects/91aabdd330f74058b7a980f075092142` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.429119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] RBAC: Authorizing `identity:delete_project(project_id=91aabdd330f74058b7a980f075092142)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:15.430997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:15.480166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91aabdd330f74058b7a980f075092142', 'request_id': 'req-772956d6-e365-4a3f-8a1f-c48dfb6b853f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:15.488043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91aabdd330f74058b7a980f075092142', 'request_id': 'req-772956d6-e365-4a3f-8a1f-c48dfb6b853f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:15.493966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91aabdd330f74058b7a980f075092142', 'request_id': 'req-772956d6-e365-4a3f-8a1f-c48dfb6b853f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:15.508259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] The token cache is being invalidate because project 91aabdd330f74058b7a980f075092142 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:56:15.508478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:15.509199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-772956d6-e365-4a3f-8a1f-c48dfb6b853f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:15.510945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1159/2311] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:56:15 2022] DELETE /identity/v3/projects/91aabdd330f74058b7a980f075092142 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:15.515737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3208e86a-9a4c-4e36-b22d-3676e680a271 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.516272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3208e86a-9a4c-4e36-b22d-3676e680a271 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.516361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3208e86a-9a4c-4e36-b22d-3676e680a271 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.538530 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3208e86a-9a4c-4e36-b22d-3676e680a271 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:15.540032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3208e86a-9a4c-4e36-b22d-3676e680a271 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:15.558857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:15.618805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1153/2312] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:15.620621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccccda2-78e9-41d1-abb4-f8b108f61862 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.620937 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccccda2-78e9-41d1-abb4-f8b108f61862 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.621175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccccda2-78e9-41d1-abb4-f8b108f61862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.649261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5ccccda2-78e9-41d1-abb4-f8b108f61862 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:15.650883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5ccccda2-78e9-41d1-abb4-f8b108f61862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:15.665809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1160/2313] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:15.666333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3208e86a-9a4c-4e36-b22d-3676e680a271 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:15.679224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3208e86a-9a4c-4e36-b22d-3676e680a271 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:15.684836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:15.685538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.685965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.686251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.687033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:56:15.687183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:15.688935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:15.707989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1154/2314] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:56:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 01:56:15.710299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccccda2-78e9-41d1-abb4-f8b108f61862 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:15.728122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccccda2-78e9-41d1-abb4-f8b108f61862 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:15.728122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:15.728122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.728122 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.728457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.728457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:15.734210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:15.755908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:15.763438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1161/2315] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:56:15 2022] POST /identity/v3/projects => generated 373 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:56:15.768547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:15.779613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbe2ce4-ec8e-44b6-bd54-c8df68ed2869 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:15.842529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:15.843180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.843460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.843730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.844403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:15.847103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:15.867687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1155/2316] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:56:15 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:15.871877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:15.882370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6642ab9-d87d-47ce-914c-aff2b00f7e2a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:15.886948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:15.887533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.887807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.888112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.888745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:15.890313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:15.900681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1162/2317] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:15.904615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:15.915171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89caf77e-e462-4271-890b-d3cc0c17a7e9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:15.919672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4908cf4-f266-4e6c-848b-349b29a542dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:15.920284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4908cf4-f266-4e6c-848b-349b29a542dc admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.920558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4908cf4-f266-4e6c-848b-349b29a542dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.920819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4908cf4-f266-4e6c-848b-349b29a542dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.921444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4908cf4-f266-4e6c-848b-349b29a542dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:15.923071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4908cf4-f266-4e6c-848b-349b29a542dc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:15.933529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1156/2318] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:15.937722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:15.947850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07650843-5a2c-4308-b1a2-9da225a2150b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:15.953288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:15.953933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:15.955233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:15.955519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] PATH_INFO: `/v3/projects/37a286c8b3eb42ae893045e96c752f86/users/1232310557b34aefb66be6df90601d9b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:15.978690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=37a286c8b3eb42ae893045e96c752f86, user_id=1232310557b34aefb66be6df90601d9b, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:15.978690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:15.989987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1163/2319] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:56:15 2022] PUT /identity/v3/projects/37a286c8b3eb42ae893045e96c752f86/users/1232310557b34aefb66be6df90601d9b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:16.000136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:16.000422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:16.000691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:16.028862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] MFA Rules not processed for user `1232310557b34aefb66be6df90601d9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:16.029971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:16.093504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1157/2320] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:16.104827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:16.106496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:16.163637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56f344-d381-48da-9eef-3aa3d29dd4fb admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:16.167229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e0cda8-1ef4-4b3e-bb84-351b53f694bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:16.167819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0cda8-1ef4-4b3e-bb84-351b53f694bb service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:16.168123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0cda8-1ef4-4b3e-bb84-351b53f694bb service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:16.168391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e0cda8-1ef4-4b3e-bb84-351b53f694bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:16.176526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e0cda8-1ef4-4b3e-bb84-351b53f694bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:16.177939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e0cda8-1ef4-4b3e-bb84-351b53f694bb service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:16.186462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1164/2321] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:16 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:24.088136 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:24.089552 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:24.108930 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:24.156569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1844b1a8-d404-401d-865e-4f0725b39645 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:24.161359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:24.161974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:24.162258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:24.162513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] PATH_INFO: `/v3/users/26fb83174b96449b85a381fcc0e9fc2b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:24.163612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26fb83174b96449b85a381fcc0e9fc2b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:24.165566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:24.211378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26fb83174b96449b85a381fcc0e9fc2b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.212038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26fb83174b96449b85a381fcc0e9fc2b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.218746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26fb83174b96449b85a381fcc0e9fc2b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.219328 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26fb83174b96449b85a381fcc0e9fc2b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.230699 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26fb83174b96449b85a381fcc0e9fc2b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.242811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1158/2322] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:56:24 2022] DELETE /identity/v3/users/26fb83174b96449b85a381fcc0e9fc2b => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:24.247449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:24.247908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:24.248029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:24.267308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:24.268518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:24.329178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1165/2323] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:24.338716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:24.340122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:24.397333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e52d03-9309-4dc6-981a-e79ed1096678 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:24.401083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d66efc2-e276-4d96-bcdf-00dacc5a8625 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:24.401678 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d66efc2-e276-4d96-bcdf-00dacc5a8625 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:24.401956 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d66efc2-e276-4d96-bcdf-00dacc5a8625 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:24.402221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d66efc2-e276-4d96-bcdf-00dacc5a8625 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:24.411828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d66efc2-e276-4d96-bcdf-00dacc5a8625 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:24.413283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d66efc2-e276-4d96-bcdf-00dacc5a8625 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:24.423232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1159/2324] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:24.702648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:24.704329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:24.718485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07684b3-d48e-4e83-8445-946cbdc9e437 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:24.724279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:24.725010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:24.725357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:24.725778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] PATH_INFO: `/v3/projects/a93da81805a145908bddc4c70d77c8ca` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:24.726878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a93da81805a145908bddc4c70d77c8ca)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:24.729005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:24.785289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a93da81805a145908bddc4c70d77c8ca', 'request_id': 'req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.792442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a93da81805a145908bddc4c70d77c8ca', 'request_id': 'req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.806872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a93da81805a145908bddc4c70d77c8ca', 'request_id': 'req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.808899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:24.834289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] The token cache is being invalidate because project a93da81805a145908bddc4c70d77c8ca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:56:24.834588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.835540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-47c1b1c8-45e3-4e1c-ae85-190ca41ad526 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:24.837612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1166/2325] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:56:24 2022] DELETE /identity/v3/projects/a93da81805a145908bddc4c70d77c8ca => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:24.842587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67946807-e517-41a8-9fa0-ecc6156b0410 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:24.842875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67946807-e517-41a8-9fa0-ecc6156b0410 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:24.843148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67946807-e517-41a8-9fa0-ecc6156b0410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:24.876831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-67946807-e517-41a8-9fa0-ecc6156b0410 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:24.878097 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-67946807-e517-41a8-9fa0-ecc6156b0410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:24.919212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:24.951988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1160/2326] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:24.960721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e04956-6b11-4eee-96e5-4370fead0979 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:24.961415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e04956-6b11-4eee-96e5-4370fead0979 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:24.962193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e04956-6b11-4eee-96e5-4370fead0979 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:24.985577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-28e04956-6b11-4eee-96e5-4370fead0979 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:24.986800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-28e04956-6b11-4eee-96e5-4370fead0979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:24.995405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1167/2327] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:24.999868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67946807-e517-41a8-9fa0-ecc6156b0410 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.010802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67946807-e517-41a8-9fa0-ecc6156b0410 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.016153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.017391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.017667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.018015 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.018717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:56:25.019011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.020694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.039347 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1161/2328] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:56:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:56:25.047763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e04956-6b11-4eee-96e5-4370fead0979 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.059562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e04956-6b11-4eee-96e5-4370fead0979 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.065931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.066563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.066845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.067119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.068007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.069885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.111974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1168/2329] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:56:25 2022] POST /identity/v3/projects => generated 385 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:25.113393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.127389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b29f3a-8c2d-490a-9ffe-3ec68110d670 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.209728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.210333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.210606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.210872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.211492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.213461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.236683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1162/2330] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:56:25 2022] POST /identity/v3/users => generated 293 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:25.240503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.251293 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1bde6ad-f78d-41bb-abbd-e90dc3d19bd7 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.257528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.258129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.258475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.259289 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.259451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.261182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.272567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1169/2331] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:56:25.276644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.287821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ddf4c0-ee03-4691-9e24-39c783b72ec6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.292912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12be6a8-6a4c-4a64-9946-a6070a27710f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.293749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12be6a8-6a4c-4a64-9946-a6070a27710f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.293972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12be6a8-6a4c-4a64-9946-a6070a27710f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.294057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12be6a8-6a4c-4a64-9946-a6070a27710f admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.294758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12be6a8-6a4c-4a64-9946-a6070a27710f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.297055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12be6a8-6a4c-4a64-9946-a6070a27710f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.308969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1163/2332] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:25.312893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.323789 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7458dedb-8719-40eb-bd3a-645e24aa798e admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.331619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.332326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.332607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.332883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] PATH_INFO: `/v3/projects/453951df68a44b51855f376845bd608f/users/3cfb3462956c4586b5ddbce0dcfd1c97/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.349875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] RBAC: Authorizing `identity:create_grant(project_id=453951df68a44b51855f376845bd608f, user_id=3cfb3462956c4586b5ddbce0dcfd1c97, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.353551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.373341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1170/2333] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:56:25 2022] PUT /identity/v3/projects/453951df68a44b51855f376845bd608f/users/3cfb3462956c4586b5ddbce0dcfd1c97/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:25.386613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86fd988-9cd8-4a4c-95e1-cfd2438462a7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.386906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86fd988-9cd8-4a4c-95e1-cfd2438462a7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.387162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86fd988-9cd8-4a4c-95e1-cfd2438462a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.443004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e86fd988-9cd8-4a4c-95e1-cfd2438462a7 None None] MFA Rules not processed for user `3cfb3462956c4586b5ddbce0dcfd1c97`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:25.444342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e86fd988-9cd8-4a4c-95e1-cfd2438462a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:25.524121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1164/2334] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:25.528594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.612079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d08831-8bda-4e7c-a482-20333ffb1387 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.616860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.617464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.617744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.618312 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.618635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.620572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.659183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1171/2335] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:56:25 2022] POST /identity/v3/projects => generated 383 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:25.663481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86fd988-9cd8-4a4c-95e1-cfd2438462a7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.676086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86fd988-9cd8-4a4c-95e1-cfd2438462a7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.709373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:25.739690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.739690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.739690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.739690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.740165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.740165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.763973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1165/2336] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:56:25 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:25.765910 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.777035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc522ab9-08cd-4b9c-9d90-901385efbff9 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.781766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.783138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.783415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.783682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.784544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.786180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.796901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1172/2337] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 01:56:25.800962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.818841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a43538-e36e-43e4-b934-84ac999a02be admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.818841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37f97b1-3c3e-407e-bfe4-50d55e2eab14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.819020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f97b1-3c3e-407e-bfe4-50d55e2eab14 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.819366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f97b1-3c3e-407e-bfe4-50d55e2eab14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.819639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37f97b1-3c3e-407e-bfe4-50d55e2eab14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.820373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37f97b1-3c3e-407e-bfe4-50d55e2eab14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.821967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37f97b1-3c3e-407e-bfe4-50d55e2eab14 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.833298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1166/2338] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:25.837388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:25.848972 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a14028-ce24-4ddf-bc2e-063133eafe0f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:25.853536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:25.854166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.854441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.854707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] PATH_INFO: `/v3/projects/4ce7893ebb4a4f969e52546edde80faa/users/958bb9953e9c4f0b8eb6c265049e4651/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.869742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ce7893ebb4a4f969e52546edde80faa, user_id=958bb9953e9c4f0b8eb6c265049e4651, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:25.872402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:25.887771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1173/2339] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:56:25 2022] PUT /identity/v3/projects/4ce7893ebb4a4f969e52546edde80faa/users/958bb9953e9c4f0b8eb6c265049e4651/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:25.902092 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd57376-f39f-4b31-b388-9bf3a0b4e2fa None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:25.902424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd57376-f39f-4b31-b388-9bf3a0b4e2fa None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:25.902636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd57376-f39f-4b31-b388-9bf3a0b4e2fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:25.932081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8cd57376-f39f-4b31-b388-9bf3a0b4e2fa None None] MFA Rules not processed for user `958bb9953e9c4f0b8eb6c265049e4651`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:25.933288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8cd57376-f39f-4b31-b388-9bf3a0b4e2fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:26.003462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1167/2340] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:25 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:56:26.008690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:26.058899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=5,keystone:DELETE=1,keystone:SELECT=94 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:26.094809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7f309-f946-439c-89a3-0cdf4d5d2ec4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:26.100813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:26.102032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.102032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.102262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.102915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:26.104943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:26.152548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1174/2341] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:56:26 2022] POST /identity/v3/projects => generated 385 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:56:26.153704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd57376-f39f-4b31-b388-9bf3a0b4e2fa None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:26.173963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd57376-f39f-4b31-b388-9bf3a0b4e2fa None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:26.240776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:26.241432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.241714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.241985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.242620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:26.244704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:26.267979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1168/2342] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:56:26 2022] POST /identity/v3/users => generated 299 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:26.272175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:26.283728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f4a3fd-9ac5-4c08-b119-5e8492ca7035 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:26.288403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:26.288997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.289273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.289536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.290389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:26.292041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:26.303555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1175/2343] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:26.307900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:26.318665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3067730e-03d7-4993-8ea0-4da15ff18145 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:26.323281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:26.323920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.324224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.324490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] PATH_INFO: `/v3/projects/fd5d6bd392b2482ea44da3b030fc7573/users/e4e3b6bd275441e592864c4ce2426f31/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.339800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd5d6bd392b2482ea44da3b030fc7573, user_id=e4e3b6bd275441e592864c4ce2426f31, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:26.342445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:26.356927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1169/2344] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:56:26 2022] PUT /identity/v3/projects/fd5d6bd392b2482ea44da3b030fc7573/users/e4e3b6bd275441e592864c4ce2426f31/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:26.362109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:26.372061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59116466-eb01-4678-bd0b-db22af8aa682 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:26.434947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4204d06-0799-4d41-b4f7-d134db00d6ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:26.435549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4204d06-0799-4d41-b4f7-d134db00d6ec admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.435822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4204d06-0799-4d41-b4f7-d134db00d6ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.436116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4204d06-0799-4d41-b4f7-d134db00d6ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.436754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4204d06-0799-4d41-b4f7-d134db00d6ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:26.438450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4204d06-0799-4d41-b4f7-d134db00d6ec admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:26.448887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1176/2345] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:26 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:26.453057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:26.465985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f69099-ec1b-4acb-a36b-a93816f306ba admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:26.471475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:26.472178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.472507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.472780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] PATH_INFO: `/v3/projects/fd5d6bd392b2482ea44da3b030fc7573/users/e4e3b6bd275441e592864c4ce2426f31/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.474672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd5d6bd392b2482ea44da3b030fc7573, user_id=e4e3b6bd275441e592864c4ce2426f31, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:26.477338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:26.480593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:26.491565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1170/2346] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:56:26 2022] PUT /identity/v3/projects/fd5d6bd392b2482ea44da3b030fc7573/users/e4e3b6bd275441e592864c4ce2426f31/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:26.502472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84868884-3b7e-4e51-ad82-0355e6259dae None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.502771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84868884-3b7e-4e51-ad82-0355e6259dae None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.503068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84868884-3b7e-4e51-ad82-0355e6259dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.535128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-84868884-3b7e-4e51-ad82-0355e6259dae None None] MFA Rules not processed for user `e4e3b6bd275441e592864c4ce2426f31`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:26.536504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-84868884-3b7e-4e51-ad82-0355e6259dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:26.619258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1177/2347] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:26 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:26.630979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:26.632831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:26.695534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23601bc7-0f1f-4d89-a923-f6a317415176 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:26.699193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:26.699790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:26.700102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:26.700374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:26.758080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:26.759512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:26.768903 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1171/2348] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:26 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:28.311791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84868884-3b7e-4e51-ad82-0355e6259dae None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:28.321978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84868884-3b7e-4e51-ad82-0355e6259dae None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:28.325501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:28.326095 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:28.326368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:28.326635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:28.357055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:28.384653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:28.386026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:28.395915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1178/2349] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:28 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:28.959573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:28.968668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523a9b52-82e7-41fa-8ad4-b223d866758b service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:28.973146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2fa5e4-bca5-492e-93c8-5c2d7c76bcaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:28.973726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2fa5e4-bca5-492e-93c8-5c2d7c76bcaf service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:28.974004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2fa5e4-bca5-492e-93c8-5c2d7c76bcaf service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:28.974305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2fa5e4-bca5-492e-93c8-5c2d7c76bcaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:28.983579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2fa5e4-bca5-492e-93c8-5c2d7c76bcaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:28.985608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2fa5e4-bca5-492e-93c8-5c2d7c76bcaf service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:28.995550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1172/2350] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:28 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:30.956320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:30.957714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:31.021713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d70f197-b6df-42f9-95b5-18e1a3bae721 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:31.026253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:31.026872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.027154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.027427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] PATH_INFO: `/v3/users/1232310557b34aefb66be6df90601d9b` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.028558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] RBAC: Authorizing `identity:delete_user(user_id=1232310557b34aefb66be6df90601d9b)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:31.030493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:31.077643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1232310557b34aefb66be6df90601d9b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.078213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1232310557b34aefb66be6df90601d9b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.079028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1232310557b34aefb66be6df90601d9b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.085845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1232310557b34aefb66be6df90601d9b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.097805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1232310557b34aefb66be6df90601d9b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.110619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1179/2351] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:56:30 2022] DELETE /identity/v3/users/1232310557b34aefb66be6df90601d9b => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:31.115050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.115505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.115688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.135777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:31.137005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:31.196588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1173/2352] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:56:31.206610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:31.208080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:31.273273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17e54b6-173b-42b6-af4f-3695349f4fcb admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:31.277447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b03713-bebf-4104-85f4-9f12295a464e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:31.278195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b03713-bebf-4104-85f4-9f12295a464e service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.278513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b03713-bebf-4104-85f4-9f12295a464e service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.278829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b03713-bebf-4104-85f4-9f12295a464e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.285748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:31.290052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b03713-bebf-4104-85f4-9f12295a464e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:31.292200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b03713-bebf-4104-85f4-9f12295a464e service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:31.305120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1180/2353] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:31.599545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:31.601207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:31.623019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16769c8b-f36b-40d6-9640-276dc2f58a38 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:31.629323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:31.631335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.632116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.632858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] PATH_INFO: `/v3/projects/37a286c8b3eb42ae893045e96c752f86` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.636786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=37a286c8b3eb42ae893045e96c752f86)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:31.638698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:31.681078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:31.702029 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37a286c8b3eb42ae893045e96c752f86', 'request_id': 'req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.709780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37a286c8b3eb42ae893045e96c752f86', 'request_id': 'req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.717698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37a286c8b3eb42ae893045e96c752f86', 'request_id': 'req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.732110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] The token cache is being invalidate because project 37a286c8b3eb42ae893045e96c752f86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:56:31.732597 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.733298 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-77d283a5-25a4-41a8-8567-fefdbdc0eb9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:31.735771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1174/2354] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:56:31 2022] DELETE /identity/v3/projects/37a286c8b3eb42ae893045e96c752f86 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:31.740579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa43ff3-2e5e-4c9f-b049-40e684b33cd5 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.741041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa43ff3-2e5e-4c9f-b049-40e684b33cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.741535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa43ff3-2e5e-4c9f-b049-40e684b33cd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.764146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-5aa43ff3-2e5e-4c9f-b049-40e684b33cd5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:31.766434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5aa43ff3-2e5e-4c9f-b049-40e684b33cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:31.843774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1181/2355] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:31.854899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbe5424-4a5e-4595-bbd7-c19edbef74b3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.855120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbe5424-4a5e-4595-bbd7-c19edbef74b3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.855377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbe5424-4a5e-4595-bbd7-c19edbef74b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.876928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ebbe5424-4a5e-4595-bbd7-c19edbef74b3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:31.879167 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ebbe5424-4a5e-4595-bbd7-c19edbef74b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:31.890640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1175/2356] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:56:31.893699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa43ff3-2e5e-4c9f-b049-40e684b33cd5 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:31.908073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa43ff3-2e5e-4c9f-b049-40e684b33cd5 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:31.915228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:31.915982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.916230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.916428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.917119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:56:31.917424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:31.919274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:31.932485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1182/2357] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:56:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:56:31.936908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbe5424-4a5e-4595-bbd7-c19edbef74b3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:31.947041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbe5424-4a5e-4595-bbd7-c19edbef74b3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:31.951927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:31.952670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:31.953042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:31.953390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:31.954713 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:31.966242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:31.996857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1176/2358] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:56:31 2022] POST /identity/v3/projects => generated 371 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:32.002078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:32.012563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50ac6cd-d7a3-4a40-8f89-5f26f71de1ae admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:32.069785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:32.070383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:32.070718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:32.070910 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:32.071713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:32.073719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:32.096755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1183/2359] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:56:32 2022] POST /identity/v3/users => generated 286 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:56:32.101105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:32.111606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f97a67-95ae-4cd7-9df7-9fed4cea63d2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:32.116901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:32.117568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:32.117990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:32.118362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:32.119217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:32.121241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:32.132686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1177/2360] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:32.136731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:32.148270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156c15a7-0537-4079-af7d-57e70b693376 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:32.156364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbad43e7-88bc-4c14-aed9-cdd0cf4078e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:32.157403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbad43e7-88bc-4c14-aed9-cdd0cf4078e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:32.157909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbad43e7-88bc-4c14-aed9-cdd0cf4078e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:32.158488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbad43e7-88bc-4c14-aed9-cdd0cf4078e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:32.159834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbad43e7-88bc-4c14-aed9-cdd0cf4078e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:32.162464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbad43e7-88bc-4c14-aed9-cdd0cf4078e3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:32.174296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1184/2361] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:32 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:32.178460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:32.189782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64c3611-ae96-438c-849a-c0d74d684e4d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:32.195350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:32.196009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:32.196292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:32.196564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] PATH_INFO: `/v3/projects/fbe6e7560b314358bd9589ac38b69320/users/a194fd36e3024b1f86837cc12ac0c2bb/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:32.212212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbe6e7560b314358bd9589ac38b69320, user_id=a194fd36e3024b1f86837cc12ac0c2bb, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:32.214906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:32.229675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1178/2362] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:56:32 2022] PUT /identity/v3/projects/fbe6e7560b314358bd9589ac38b69320/users/a194fd36e3024b1f86837cc12ac0c2bb/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:32.240063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:32.240347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:32.240611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:32.270891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] MFA Rules not processed for user `a194fd36e3024b1f86837cc12ac0c2bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:32.272153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:32.319452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:32.337985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1185/2363] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:32 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:32.349109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:32.350774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:32.407397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72be0ca4-c713-4780-8108-1d0bab55591e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:32.410968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e72532-094a-4bed-b264-4f5fd1a9f6d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:32.411560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e72532-094a-4bed-b264-4f5fd1a9f6d4 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:32.411836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e72532-094a-4bed-b264-4f5fd1a9f6d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:32.412147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e72532-094a-4bed-b264-4f5fd1a9f6d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:32.420326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e72532-094a-4bed-b264-4f5fd1a9f6d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:32.421743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e72532-094a-4bed-b264-4f5fd1a9f6d4 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:32.430855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1179/2364] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:32 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:33.000099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:33.001586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:33.069374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03b15e5-0d2f-4e21-8398-420cea63be56 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:33.074644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:33.075273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:33.075552 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:33.075822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] PATH_INFO: `/v3/users/3cfb3462956c4586b5ddbce0dcfd1c97` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:33.077033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3cfb3462956c4586b5ddbce0dcfd1c97)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:33.078971 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:33.139553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3cfb3462956c4586b5ddbce0dcfd1c97'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.140362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3cfb3462956c4586b5ddbce0dcfd1c97'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.141118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3cfb3462956c4586b5ddbce0dcfd1c97'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.148766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3cfb3462956c4586b5ddbce0dcfd1c97'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.163754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3cfb3462956c4586b5ddbce0dcfd1c97'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.180469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1186/2365] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:56:32 2022] DELETE /identity/v3/users/3cfb3462956c4586b5ddbce0dcfd1c97 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:56:33.185262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:33.185847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:33.185847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:33.206886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:33.208362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:33.263681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:33.273929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1180/2366] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:33.283636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:33.285068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:33.341757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14053913-53b0-48e6-87f7-ec8663e6d2b6 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:33.345422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:33.346019 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:33.346297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:33.346567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:33.355704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:33.357160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:33.367066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1187/2367] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:56:33.631844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:33.633433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:33.645824 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea20f6b-9ba5-4873-8d3f-3bec91848ec5 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:33.650420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:33.651042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:33.651317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:33.651584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] PATH_INFO: `/v3/projects/453951df68a44b51855f376845bd608f` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:33.652723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] RBAC: Authorizing `identity:delete_project(project_id=453951df68a44b51855f376845bd608f)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:33.654712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:33.703473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '453951df68a44b51855f376845bd608f', 'request_id': 'req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.709708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '453951df68a44b51855f376845bd608f', 'request_id': 'req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.715125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '453951df68a44b51855f376845bd608f', 'request_id': 'req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.728355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] The token cache is being invalidate because project 453951df68a44b51855f376845bd608f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:56:33.728601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.730007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f2291c6e-9bc7-4605-aa95-e2ffe13f2439 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.730723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1181/2368] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:56:33 2022] DELETE /identity/v3/projects/453951df68a44b51855f376845bd608f => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:33.735617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:33.747367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01252460-8956-4abd-84eb-0e7dbe55ca47 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:33.757546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:33.792791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:33.793070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:33.793338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:33.817772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:33.818465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:33.818746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:33.819047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] PATH_INFO: `/v3/users/958bb9953e9c4f0b8eb6c265049e4651` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:33.820274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] RBAC: Authorizing `identity:delete_user(user_id=958bb9953e9c4f0b8eb6c265049e4651)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:33.822302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:33.824098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] MFA Rules not processed for user `a194fd36e3024b1f86837cc12ac0c2bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:33.825825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:33.878101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '958bb9953e9c4f0b8eb6c265049e4651'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.879151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '958bb9953e9c4f0b8eb6c265049e4651'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.880636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '958bb9953e9c4f0b8eb6c265049e4651'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.889494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '958bb9953e9c4f0b8eb6c265049e4651'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.892034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1182/2369] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:33 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:56:33.904852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '958bb9953e9c4f0b8eb6c265049e4651'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:33.923008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1188/2370] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:56:33 2022] DELETE /identity/v3/users/958bb9953e9c4f0b8eb6c265049e4651 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:34.512674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:34.514381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:34.547372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596b415e-fbe7-4e58-97ec-f20d6545de8d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:34.680031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:34.680869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:34.681329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:34.681926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] PATH_INFO: `/v3/projects/4ce7893ebb4a4f969e52546edde80faa` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:34.683213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ce7893ebb4a4f969e52546edde80faa)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:34.685458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:34.786256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:34.805168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ce7893ebb4a4f969e52546edde80faa', 'request_id': 'req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:34.815910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ce7893ebb4a4f969e52546edde80faa', 'request_id': 'req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:34.823977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ce7893ebb4a4f969e52546edde80faa', 'request_id': 'req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:34.839521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] The token cache is being invalidate because project 4ce7893ebb4a4f969e52546edde80faa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:56:34.839841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:34.840535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:34.842214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1183/2371] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:56:34 2022] DELETE /identity/v3/projects/4ce7893ebb4a4f969e52546edde80faa => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:34.847458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:34.859091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0c37a-13ee-4c54-8357-95e5a7d9978a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:34.940670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:34.941469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:34.941883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:34.942220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] PATH_INFO: `/v3/users/e4e3b6bd275441e592864c4ce2426f31` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:34.943512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4e3b6bd275441e592864c4ce2426f31)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:34.945617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:34.984660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:34.999199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4e3b6bd275441e592864c4ce2426f31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:34.999878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4e3b6bd275441e592864c4ce2426f31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.001047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4e3b6bd275441e592864c4ce2426f31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.008947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4e3b6bd275441e592864c4ce2426f31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.026403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-72f1976f-186b-4309-9edf-8ff3a9c3f392 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4e3b6bd275441e592864c4ce2426f31'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.040646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1189/2372] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:56:34 2022] DELETE /identity/v3/users/e4e3b6bd275441e592864c4ce2426f31 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:56:35.315288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:35.316818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:35.334493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b508c7-b5ea-4d1e-bcd7-021b89c141d1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:35.393789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:35.394414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.394693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.394968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] PATH_INFO: `/v3/projects/fd5d6bd392b2482ea44da3b030fc7573` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.396125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd5d6bd392b2482ea44da3b030fc7573)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:35.398062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:35.454615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd5d6bd392b2482ea44da3b030fc7573', 'request_id': 'req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.462149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd5d6bd392b2482ea44da3b030fc7573', 'request_id': 'req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.469262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd5d6bd392b2482ea44da3b030fc7573', 'request_id': 'req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.482826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] The token cache is being invalidate because project fd5d6bd392b2482ea44da3b030fc7573 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:56:35.483102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.483742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-7ffdfae6-7748-4549-8d7b-5af611a4ba9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:56:35.489872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1184/2373] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:56:35 2022] DELETE /identity/v3/projects/fd5d6bd392b2482ea44da3b030fc7573 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:56:35.491364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f640d6-ef07-471b-a3bd-3ce9bca785ff None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.491692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f640d6-ef07-471b-a3bd-3ce9bca785ff None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.491965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f640d6-ef07-471b-a3bd-3ce9bca785ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.515076 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-47f640d6-ef07-471b-a3bd-3ce9bca785ff None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:35.517049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-47f640d6-ef07-471b-a3bd-3ce9bca785ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:35.587756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1190/2374] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:35.596066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb144d6b-49cf-461d-b89b-186b02c4766c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.596675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb144d6b-49cf-461d-b89b-186b02c4766c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.596675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb144d6b-49cf-461d-b89b-186b02c4766c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.619003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-eb144d6b-49cf-461d-b89b-186b02c4766c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:35.620354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-eb144d6b-49cf-461d-b89b-186b02c4766c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:35.629517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1185/2375] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:35.633757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f640d6-ef07-471b-a3bd-3ce9bca785ff None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:35.643661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f640d6-ef07-471b-a3bd-3ce9bca785ff None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:35.648274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:35.648863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.649144 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.649416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.650064 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 01:56:35.650368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:35.651960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:35.664451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1191/2376] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 01:56:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 01:56:35.670766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb144d6b-49cf-461d-b89b-186b02c4766c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:35.680745 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb144d6b-49cf-461d-b89b-186b02c4766c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:35.685621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:35.686223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.686499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.686766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.687416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:35.689386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:35.728225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1186/2377] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 01:56:35 2022] POST /identity/v3/projects => generated 391 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 01:56:35.732481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:35.745314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed5c012-863e-456f-b176-c017f97dc19a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:35.831212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:35.831903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.832221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.832492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.833127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:35.835134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:35.862600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1192/2378] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 01:56:35 2022] POST /identity/v3/users => generated 296 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 01:56:35.866326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:35.877706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe061f8-3076-4113-9f70-9183f01feff6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:35.883131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:35.883841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.884144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.884415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.885100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:35.886799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:35.897538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1187/2379] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:35.901643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:35.912629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17785322-6499-4fdd-831a-1013df96abdc admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:35.917311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334b99d-90a8-4cf2-857e-383c562e47cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:35.917904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334b99d-90a8-4cf2-857e-383c562e47cc admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.918185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334b99d-90a8-4cf2-857e-383c562e47cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.918453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334b99d-90a8-4cf2-857e-383c562e47cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.919260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f334b99d-90a8-4cf2-857e-383c562e47cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:35.920887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f334b99d-90a8-4cf2-857e-383c562e47cc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:35.932421 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1193/2380] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 01:56:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 01:56:35.936599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:35.946750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32105c3b-02eb-4632-a2f3-d1141846c333 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:35.951393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:35.952071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.952353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.952626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] PATH_INFO: `/v3/projects/6d6d53faedb644b18997ae395403234f/users/3af1fb0ddf5348a48efb90edca5f007c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:35.955963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:35.969865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d6d53faedb644b18997ae395403234f, user_id=3af1fb0ddf5348a48efb90edca5f007c, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:35.972540 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:35.985822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1188/2381] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 01:56:35 2022] PUT /identity/v3/projects/6d6d53faedb644b18997ae395403234f/users/3af1fb0ddf5348a48efb90edca5f007c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:56:35.995909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627aff47-88e2-46e0-afb0-df2d1ad96da0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:35.996261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627aff47-88e2-46e0-afb0-df2d1ad96da0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:35.996544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627aff47-88e2-46e0-afb0-df2d1ad96da0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:36.025210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-627aff47-88e2-46e0-afb0-df2d1ad96da0 None None] MFA Rules not processed for user `3af1fb0ddf5348a48efb90edca5f007c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:36.026421 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-627aff47-88e2-46e0-afb0-df2d1ad96da0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:36.030356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:36.092105 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1194/2382] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:56:35 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:36.103632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:36.105354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:36.162457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1064a39a-fcc8-401e-b0d1-87f7c4736fd6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:36.166099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:36.166680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:36.166961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:36.167246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:36.175622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:36.177066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:36.186450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1189/2383] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:56:36 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:56:46.089508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=25 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:46.182582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=32,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:48.241664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:48.242398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:48.242398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:48.262566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:56:48.264198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:48.324383 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1195/2384] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:56:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:56:48.368263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:56:48.370495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:56:48.450177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4436395-1a3e-4ddb-be91-897a1e0cd56a service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:56:48.455251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-132c12ea-8359-44c0-aa2f-a3f2d0d6e5a2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:56:48.455848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132c12ea-8359-44c0-aa2f-a3f2d0d6e5a2 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:56:48.456154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132c12ea-8359-44c0-aa2f-a3f2d0d6e5a2 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:56:48.456424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132c12ea-8359-44c0-aa2f-a3f2d0d6e5a2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:56:48.464979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132c12ea-8359-44c0-aa2f-a3f2d0d6e5a2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:56:48.466578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-132c12ea-8359-44c0-aa2f-a3f2d0d6e5a2 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:56:48.475560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1190/2385] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:56:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:56:58.317415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=25 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:56:58.472309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=32 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:06.190109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:06.192375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:06.266269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8c41db-86ff-438d-a6fe-f39e001a1617 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:06.272382 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:06.273024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:06.273306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:06.273573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:06.275280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:06.289193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:06.290965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d783c197-05b6-4605-8006-fd435fd4767b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:06.291452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d783c197-05b6-4605-8006-fd435fd4767b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:06.291546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d783c197-05b6-4605-8006-fd435fd4767b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:06.291678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:06.302374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1196/2386] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:57:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:57:06.314284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d783c197-05b6-4605-8006-fd435fd4767b None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:06.315576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d783c197-05b6-4605-8006-fd435fd4767b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:06.324437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1191/2387] 10.222.0.42 () {58 vars in 967 bytes} [Fri Sep 16 01:57:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:06.339131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:06.350791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6be00c0-40cc-48a3-93cd-575545567421 service placement] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:06.355811 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3c7824-7025-4f6a-ba7d-dca6b21217c8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:06.356438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c7824-7025-4f6a-ba7d-dca6b21217c8 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:06.356747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c7824-7025-4f6a-ba7d-dca6b21217c8 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:06.357015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3c7824-7025-4f6a-ba7d-dca6b21217c8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:06.365055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3c7824-7025-4f6a-ba7d-dca6b21217c8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:06.366621 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3c7824-7025-4f6a-ba7d-dca6b21217c8 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:06.374865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1197/2388] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:57:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:57:06.637795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d783c197-05b6-4605-8006-fd435fd4767b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:06.713877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d783c197-05b6-4605-8006-fd435fd4767b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:06.719065 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:06.719661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:06.720165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:06.720532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] PATH_INFO: `/v3/users/3af1fb0ddf5348a48efb90edca5f007c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:06.721744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3af1fb0ddf5348a48efb90edca5f007c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:06.723856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:06.778583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3af1fb0ddf5348a48efb90edca5f007c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:06.779228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3af1fb0ddf5348a48efb90edca5f007c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:06.787622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3af1fb0ddf5348a48efb90edca5f007c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:06.788306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3af1fb0ddf5348a48efb90edca5f007c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:06.803093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3af1fb0ddf5348a48efb90edca5f007c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:06.819276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1192/2389] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:57:06 2022] DELETE /identity/v3/users/3af1fb0ddf5348a48efb90edca5f007c => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:57:06.824018 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:06.824220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:06.824564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:06.844924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:06.846131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:06.907279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1198/2390] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 01:57:06.917184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:06.918585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:06.976403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10917a1d-654f-40ce-8a14-8bfd825faa51 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:06.980052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:06.980648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:06.980922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:06.981186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:06.990714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:06.992215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:07.002290 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1193/2391] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:57:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:57:07.328818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:07.331121 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:07.346730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c10e4c-bdf0-462d-a822-e1fab9b14b30 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:07.353824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:07.354934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:07.355550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:07.356598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] PATH_INFO: `/v3/projects/6d6d53faedb644b18997ae395403234f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:07.358788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d6d53faedb644b18997ae395403234f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:07.376068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:07.440998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d6d53faedb644b18997ae395403234f', 'request_id': 'req-fdf3d33f-bf15-4409-a84a-16be0a61805d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:07.450048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d6d53faedb644b18997ae395403234f', 'request_id': 'req-fdf3d33f-bf15-4409-a84a-16be0a61805d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:07.456932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d6d53faedb644b18997ae395403234f', 'request_id': 'req-fdf3d33f-bf15-4409-a84a-16be0a61805d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:07.468805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:07.474085 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] The token cache is being invalidate because project 6d6d53faedb644b18997ae395403234f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:57:07.475195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:07.475999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fdf3d33f-bf15-4409-a84a-16be0a61805d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:07.477808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1199/2392] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:57:07 2022] DELETE /identity/v3/projects/6d6d53faedb644b18997ae395403234f => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 01:57:08.422099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:08.424134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:08.444005 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38c1167-7781-4f3b-97d5-83ed28772129 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:08.457466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:08.497767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:08.498282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:08.498282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:08.513033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:08.513667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:08.513948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:08.514762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] PATH_INFO: `/v3/users/47a422989ce94168a042135234bea46b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:08.515907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=47a422989ce94168a042135234bea46b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:08.517867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:08.518392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] MFA Rules not processed for user `7eb6ab0644bf4fa59d7bba232d6ccb3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:08.519606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:08.568592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47a422989ce94168a042135234bea46b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.569160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47a422989ce94168a042135234bea46b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.576210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47a422989ce94168a042135234bea46b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.577385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47a422989ce94168a042135234bea46b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.581208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1200/2393] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:57:08 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:08.589750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87200088-5c5f-4b5b-8e53-72857cc6ef5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47a422989ce94168a042135234bea46b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.603388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1194/2394] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:57:08 2022] DELETE /identity/v3/users/47a422989ce94168a042135234bea46b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:57:08.788779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:08.790273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:08.819408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04c2846-0664-4e3f-9189-495fe3e295e9 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:08.898257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:08.899015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:08.899617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:08.900103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] PATH_INFO: `/v3/projects/3a0380c941f644a794f51e1f575faf00` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:08.901581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a0380c941f644a794f51e1f575faf00)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:08.903661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:08.964590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a0380c941f644a794f51e1f575faf00', 'request_id': 'req-970be71f-4e5b-43ca-9664-897cdd492371'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.974630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a0380c941f644a794f51e1f575faf00', 'request_id': 'req-970be71f-4e5b-43ca-9664-897cdd492371'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.983615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a0380c941f644a794f51e1f575faf00', 'request_id': 'req-970be71f-4e5b-43ca-9664-897cdd492371'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.997501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] The token cache is being invalidate because project 3a0380c941f644a794f51e1f575faf00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:57:08.997791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:08.998426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:09.000066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1201/2395] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:57:08 2022] DELETE /identity/v3/projects/3a0380c941f644a794f51e1f575faf00 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:57:14.187852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112168b5-f12a-4ab4-b364-c0137425e815 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:14.188340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112168b5-f12a-4ab4-b364-c0137425e815 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:14.188434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112168b5-f12a-4ab4-b364-c0137425e815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:14.210566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-112168b5-f12a-4ab4-b364-c0137425e815 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:14.211756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-112168b5-f12a-4ab4-b364-c0137425e815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:14.267008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1195/2396] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:57:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:14.279403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:14.281681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:14.322525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:14.352582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970be71f-4e5b-43ca-9664-897cdd492371 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:14.357575 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:14.358731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:14.358999 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:14.359263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:14.367183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:14.368781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:14.377416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1202/2397] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:57:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:57:14.887876 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc8bc9-e55a-4e66-b12c-b9ebaae2026d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:14.888188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc8bc9-e55a-4e66-b12c-b9ebaae2026d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:14.888457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc8bc9-e55a-4e66-b12c-b9ebaae2026d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:14.907498 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7bfc8bc9-e55a-4e66-b12c-b9ebaae2026d None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:14.908705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7bfc8bc9-e55a-4e66-b12c-b9ebaae2026d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:14.915727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1196/2398] 10.222.0.42 () {58 vars in 967 bytes} [Fri Sep 16 01:57:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:14.928439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:14.938894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600c9554-739c-49bd-b1d3-ecb1573e0c41 service swift] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:14.943886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:14.944564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:14.944845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:14.945115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:14.953068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:14.954624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:14.962835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1203/2399] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:57:14 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 01:57:15.214133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c64e6f-e438-4c31-975b-446166460650 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:15.214613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c64e6f-e438-4c31-975b-446166460650 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:15.214724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c64e6f-e438-4c31-975b-446166460650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:15.234488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-93c64e6f-e438-4c31-975b-446166460650 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:15.235656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-93c64e6f-e438-4c31-975b-446166460650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:15.242644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1197/2400] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:57:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:15.255362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:15.266367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3892a144-f4ef-4ed0-abe6-57f7297a04a2 service swift] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:15.271300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:15.271908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:15.272217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:15.272504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:15.280457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:15.282044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:15.289967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1204/2401] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:57:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 01:57:15.626987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:15.627416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:15.627809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:15.649722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:15.650911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:15.658589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1198/2402] 10.222.0.42 () {58 vars in 967 bytes} [Fri Sep 16 01:57:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:15.672040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:15.682743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a89491-168c-4b0f-af8f-8c7fea570aa0 service swift] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:15.687704 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d014ee5-9d9f-447c-8a31-ae4a1f23fd01 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:15.688328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d014ee5-9d9f-447c-8a31-ae4a1f23fd01 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:15.688608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d014ee5-9d9f-447c-8a31-ae4a1f23fd01 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:15.688879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d014ee5-9d9f-447c-8a31-ae4a1f23fd01 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:15.697174 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d014ee5-9d9f-447c-8a31-ae4a1f23fd01 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:15.698726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d014ee5-9d9f-447c-8a31-ae4a1f23fd01 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:15.707084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1205/2403] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:57:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 01:57:17.850985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:17.852415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:17.874891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:17.919310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-018e1f1e-a2c1-4b3d-90f6-133ad980c09c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:17.923904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:17.924572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:17.924843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:17.925108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] PATH_INFO: `/v3/users/a194fd36e3024b1f86837cc12ac0c2bb` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:17.926217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a194fd36e3024b1f86837cc12ac0c2bb)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:17.928150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:17.974911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a194fd36e3024b1f86837cc12ac0c2bb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:17.975513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a194fd36e3024b1f86837cc12ac0c2bb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:17.984041 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a194fd36e3024b1f86837cc12ac0c2bb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:17.984694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a194fd36e3024b1f86837cc12ac0c2bb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:17.997004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a194fd36e3024b1f86837cc12ac0c2bb'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:18.011398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1199/2404] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 01:57:17 2022] DELETE /identity/v3/users/a194fd36e3024b1f86837cc12ac0c2bb => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:57:18.016675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:18.017153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:18.017385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:18.038193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:18.039410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:18.100953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1206/2405] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:57:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:18.117598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:18.119020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:18.185004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eb581d-a8c7-4cf6-a34f-c53bed4bafe9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:18.189850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc091bb1-c23c-427c-8c11-ce53fed97165 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:18.191640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc091bb1-c23c-427c-8c11-ce53fed97165 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:18.192056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc091bb1-c23c-427c-8c11-ce53fed97165 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:18.192555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc091bb1-c23c-427c-8c11-ce53fed97165 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:18.203405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc091bb1-c23c-427c-8c11-ce53fed97165 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:18.204961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc091bb1-c23c-427c-8c11-ce53fed97165 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:18.215625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1200/2406] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:57:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:57:18.389889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:18.391300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:18.406807 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f754d69d-92ed-4037-8741-563fe6c6c1e0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:18.412562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:18.413178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:18.413502 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:18.413726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] PATH_INFO: `/v3/projects/fbe6e7560b314358bd9589ac38b69320` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:18.414843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbe6e7560b314358bd9589ac38b69320)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:18.417504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:18.477957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbe6e7560b314358bd9589ac38b69320', 'request_id': 'req-2a58ac78-7d69-46ec-8786-76af2d7f5d12'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:18.485114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbe6e7560b314358bd9589ac38b69320', 'request_id': 'req-2a58ac78-7d69-46ec-8786-76af2d7f5d12'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:18.491305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbe6e7560b314358bd9589ac38b69320', 'request_id': 'req-2a58ac78-7d69-46ec-8786-76af2d7f5d12'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:18.504913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] The token cache is being invalidate because project fbe6e7560b314358bd9589ac38b69320 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 01:57:18.505202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:18.506131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 01:57:18.508726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1207/2407] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 01:57:18 2022] DELETE /identity/v3/projects/fbe6e7560b314358bd9589ac38b69320 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 01:57:28.211220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=70,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:28.501437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:45.405385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe04120a-df77-403f-a4c5-158841db8ddf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:45.405873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe04120a-df77-403f-a4c5-158841db8ddf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:45.405981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe04120a-df77-403f-a4c5-158841db8ddf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:45.425292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-fe04120a-df77-403f-a4c5-158841db8ddf None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:57:45.426462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fe04120a-df77-403f-a4c5-158841db8ddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:45.480560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1201/2408] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:57:45 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:57:45.494069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:57:45.495769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:57:45.566120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a58ac78-7d69-46ec-8786-76af2d7f5d12 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:57:45.572102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc5139c-4854-48d1-ab9e-76cb05512425 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:57:45.572700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5139c-4854-48d1-ab9e-76cb05512425 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:57:45.572973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5139c-4854-48d1-ab9e-76cb05512425 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:57:45.573238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc5139c-4854-48d1-ab9e-76cb05512425 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:57:45.583809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc5139c-4854-48d1-ab9e-76cb05512425 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:57:45.585447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc5139c-4854-48d1-ab9e-76cb05512425 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:57:45.594150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1208/2409] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:57:45 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:57:55.473781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=25 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:57:55.590326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=32 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:58:00.308380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769c8b98-ee87-4b88-b1b7-3b3466f7eaf2 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.308931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769c8b98-ee87-4b88-b1b7-3b3466f7eaf2 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.308931 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769c8b98-ee87-4b88-b1b7-3b3466f7eaf2 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.310915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1202/2410] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:58:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 01:58:00.317369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91abbdb4-1c1a-4533-8a04-5033d5f4c742 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.317858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91abbdb4-1c1a-4533-8a04-5033d5f4c742 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.317968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91abbdb4-1c1a-4533-8a04-5033d5f4c742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.338058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-91abbdb4-1c1a-4533-8a04-5033d5f4c742 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:58:00.339280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-91abbdb4-1c1a-4533-8a04-5033d5f4c742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:00.409583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1209/2411] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:58:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:58:00.424434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769c8b98-ee87-4b88-b1b7-3b3466f7eaf2 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:58:00.426164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-769c8b98-ee87-4b88-b1b7-3b3466f7eaf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:00.486412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769c8b98-ee87-4b88-b1b7-3b3466f7eaf2 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:58:00.489975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e18c67-21e0-473a-8630-73702bf3a60d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:58:00.490595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e18c67-21e0-473a-8630-73702bf3a60d service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.490875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e18c67-21e0-473a-8630-73702bf3a60d service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.491171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e18c67-21e0-473a-8630-73702bf3a60d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.501828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e18c67-21e0-473a-8630-73702bf3a60d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:58:00.503507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e18c67-21e0-473a-8630-73702bf3a60d service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:58:00.514126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1203/2412] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:58:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:58:00.582849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79126f0e-622a-4ad8-8523-6215d4d36f9f None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.583045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79126f0e-622a-4ad8-8523-6215d4d36f9f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.583326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79126f0e-622a-4ad8-8523-6215d4d36f9f None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.584710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1210/2413] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:58:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:58:00.589367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595e8bd9-fd04-4173-b512-1a74fdc3f049 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.589654 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595e8bd9-fd04-4173-b512-1a74fdc3f049 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.589928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595e8bd9-fd04-4173-b512-1a74fdc3f049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.611303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-595e8bd9-fd04-4173-b512-1a74fdc3f049 None None] MFA Rules not processed for user `6a343a260fa2415e84caf959933a6c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:58:00.612529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-595e8bd9-fd04-4173-b512-1a74fdc3f049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:00.663881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1204/2414] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:58:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:58:00.676538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79126f0e-622a-4ad8-8523-6215d4d36f9f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:58:00.678399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-79126f0e-622a-4ad8-8523-6215d4d36f9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:00.692321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79126f0e-622a-4ad8-8523-6215d4d36f9f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:58:00.695873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e0bfe8-2ffb-42f9-98c5-21dd1c05c1ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:58:00.696572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0bfe8-2ffb-42f9-98c5-21dd1c05c1ec service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.696849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0bfe8-2ffb-42f9-98c5-21dd1c05c1ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.697117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e0bfe8-2ffb-42f9-98c5-21dd1c05c1ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.705104 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e0bfe8-2ffb-42f9-98c5-21dd1c05c1ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:58:00.706609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e0bfe8-2ffb-42f9-98c5-21dd1c05c1ec service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:58:00.714295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1211/2415] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:58:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:58:00.798019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5076a0-110b-4e79-93ff-b92521b2a545 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.798385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5076a0-110b-4e79-93ff-b92521b2a545 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.798728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5076a0-110b-4e79-93ff-b92521b2a545 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.800475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1205/2416] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 01:58:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:58:00.806791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada5f7cb-7050-4c34-9c31-7116591e423c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.807077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada5f7cb-7050-4c34-9c31-7116591e423c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.807346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada5f7cb-7050-4c34-9c31-7116591e423c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.829137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-ada5f7cb-7050-4c34-9c31-7116591e423c None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:58:00.830323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ada5f7cb-7050-4c34-9c31-7116591e423c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:00.839596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1212/2417] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 01:58:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:58:00.849939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5076a0-110b-4e79-93ff-b92521b2a545 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:58:00.861288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5076a0-110b-4e79-93ff-b92521b2a545 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:58:00.864853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:58:00.865436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:00.865712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:00.865980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:00.876979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:58:00.878656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:58:00.889374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1206/2418] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:58:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:58:04.134450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f7d35-3f83-4b0d-b2c1-c8e460ab1378 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:04.134948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f7d35-3f83-4b0d-b2c1-c8e460ab1378 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:04.135188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4f7d35-3f83-4b0d-b2c1-c8e460ab1378 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:04.155161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-da4f7d35-3f83-4b0d-b2c1-c8e460ab1378 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:58:04.156449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-da4f7d35-3f83-4b0d-b2c1-c8e460ab1378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:04.163779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1213/2419] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 01:58:04 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:58:04.176495 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:58:04.188887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf3942-022a-42e0-8fd6-4580a48a09a1 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:58:04.193904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:58:04.194568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:04.194916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:04.195360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:04.204336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:58:04.206030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:58:04.213889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1207/2420] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:58:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:58:14.149501 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=43 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:58:14.210743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=61 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:58:32.754409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:32.754897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:32.754995 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:32.774211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:58:32.775412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:32.782794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1214/2421] 10.222.0.42 () {58 vars in 967 bytes} [Fri Sep 16 01:58:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:58:32.795214 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:58:32.806404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65689b6a-452f-432b-9563-2d96809da29f service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:58:32.812299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:58:32.812896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:32.813171 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:32.813440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:32.821634 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:58:32.823213 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:58:32.831789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1208/2422] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 01:58:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 01:58:42.769083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:58:42.828685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:58:51.962208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:58:51.963624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:52.022094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a240b45-a658-4c0a-83d0-2db71b1179d0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:58:52.025881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:58:52.026485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:52.026829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:52.027592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:52.029423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:52.043013 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:58:52.044484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:58:52.055338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1215/2423] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:58:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:58:59.914500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:58:59.915905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:59.928042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bbd10e-032b-4be6-8b7d-c7c49f4acedf service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:58:59.931512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:58:59.932123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:58:59.932398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:58:59.932662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:58:59.934275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:58:59.947265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:58:59.948679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:58:59.959992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1209/2424] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:58:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:59:02.050554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=26 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:03.881094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:03.882531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:03.896423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6bb740-845d-45c4-a370-029bb96eb2b9 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:03.901858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:03.902469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:03.902749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:03.903020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:03.904902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:03.962376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:03.963871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:03.973368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1216/2425] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:03 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:59:09.955260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:10.704275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:10.715262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d16e44-eaed-40fe-811b-613251f62382 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:10.721335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f5a7ad-57fe-457b-9382-cd4db44933e1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:10.721917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5a7ad-57fe-457b-9382-cd4db44933e1 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:10.722192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5a7ad-57fe-457b-9382-cd4db44933e1 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:10.722465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5a7ad-57fe-457b-9382-cd4db44933e1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:10.724176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e8f5a7ad-57fe-457b-9382-cd4db44933e1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:10.782734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f5a7ad-57fe-457b-9382-cd4db44933e1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:10.784464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f5a7ad-57fe-457b-9382-cd4db44933e1 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:10.793743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1210/2426] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:10 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:59:13.968945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=28 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:17.519318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:17.530862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1501ec1e-cd41-45cb-adf8-497117e10cb1 service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:17.534645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:17.535271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:17.535530 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:17.535831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:17.537639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:17.549986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:17.551345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:17.559433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1217/2427] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:59:20.789565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=28 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:25.131614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13291f3-8dd1-4a02-9779-9366884222e6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:25.132277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13291f3-8dd1-4a02-9779-9366884222e6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:25.132277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13291f3-8dd1-4a02-9779-9366884222e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:25.150992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d13291f3-8dd1-4a02-9779-9366884222e6 None None] MFA Rules not processed for user `7eb6ab0644bf4fa59d7bba232d6ccb3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:59:25.152215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d13291f3-8dd1-4a02-9779-9366884222e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:25.159896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1211/2428] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:59:25 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:59:27.555839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:30.328233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:30.343413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380f5493-6992-49bf-89a6-56ca561a810c service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:30.349703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40ae5ba-592a-475d-a8d0-d97598fcd33a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:30.350357 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ae5ba-592a-475d-a8d0-d97598fcd33a service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:30.350636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ae5ba-592a-475d-a8d0-d97598fcd33a service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:30.350897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40ae5ba-592a-475d-a8d0-d97598fcd33a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:30.352730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f40ae5ba-592a-475d-a8d0-d97598fcd33a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:30.366261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40ae5ba-592a-475d-a8d0-d97598fcd33a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:30.367888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40ae5ba-592a-475d-a8d0-d97598fcd33a service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:30.379392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1218/2429] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:59:35.145489 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:38.968381 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13291f3-8dd1-4a02-9779-9366884222e6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:38.981197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13291f3-8dd1-4a02-9779-9366884222e6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:38.988728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:38.989321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:38.989595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:38.989859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:38.991569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:39.052937 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:39.054597 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:39.065018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1212/2430] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:38 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:59:40.374565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:47.375126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3c70d-b70f-4998-871e-090ae4e39268 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:47.375748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3c70d-b70f-4998-871e-090ae4e39268 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:47.375748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf3c70d-b70f-4998-871e-090ae4e39268 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:47.395023 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4bf3c70d-b70f-4998-871e-090ae4e39268 None None] MFA Rules not processed for user `dd95ef263d15460f9cca5a7557cbee2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:59:47.396230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4bf3c70d-b70f-4998-871e-090ae4e39268 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:47.402945 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1219/2431] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 01:59:47 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:59:47.709294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:47.710836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:47.722625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528e1e57-7ead-483d-ae9c-05c3c17e7a66 service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:47.726452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfe0466-bd0f-4e84-b2aa-2217924a4d94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:47.727099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe0466-bd0f-4e84-b2aa-2217924a4d94 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:47.727375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe0466-bd0f-4e84-b2aa-2217924a4d94 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:47.727639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe0466-bd0f-4e84-b2aa-2217924a4d94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:47.729295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cdfe0466-bd0f-4e84-b2aa-2217924a4d94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:47.739603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfe0466-bd0f-4e84-b2aa-2217924a4d94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:47.741209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfe0466-bd0f-4e84-b2aa-2217924a4d94 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:47.749729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1213/2432] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:47 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 01:59:48.063959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a5e72d-e296-4351-9cce-08c89e966f7c None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:48.064340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a5e72d-e296-4351-9cce-08c89e966f7c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:48.064609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a5e72d-e296-4351-9cce-08c89e966f7c None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:48.065962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1220/2433] 10.222.0.42 () {54 vars in 961 bytes} [Fri Sep 16 01:59:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 01:59:48.070829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:48.071178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:48.071508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:48.092791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] MFA Rules not processed for user `b4311561ef9743988bdf64a039c0628f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 01:59:48.093935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:48.156979 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1214/2434] 10.222.0.42 () {58 vars in 1058 bytes} [Fri Sep 16 01:59:48 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 01:59:48.164892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a5e72d-e296-4351-9cce-08c89e966f7c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:48.178903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a5e72d-e296-4351-9cce-08c89e966f7c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:48.184090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221c1ae6-74fe-4d90-b9c1-b7d025c245c4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:48.184667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221c1ae6-74fe-4d90-b9c1-b7d025c245c4 service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:48.184942 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221c1ae6-74fe-4d90-b9c1-b7d025c245c4 service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:48.185856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221c1ae6-74fe-4d90-b9c1-b7d025c245c4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:48.187875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-221c1ae6-74fe-4d90-b9c1-b7d025c245c4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:48.198968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221c1ae6-74fe-4d90-b9c1-b7d025c245c4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:48.200558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221c1ae6-74fe-4d90-b9c1-b7d025c245c4 service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:48.210532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1221/2435] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 01:59:58.148404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=63 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:58.206399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=9 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 01:59:58.874341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 01:59:58.875828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:58.895275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116cac0a-f004-4e78-9f4e-52d1c0a3cfdc None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 01:59:58.901188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 01:59:58.901797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 01:59:58.902074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 01:59:58.902342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 01:59:58.904697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 01:59:58.917890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 01:59:58.920102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 01:59:58.931838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1215/2436] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 01:59:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:00:08.926763 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:24.022522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b00619b-d385-4ea6-a3b8-79eca15740b6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:24.023119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b00619b-d385-4ea6-a3b8-79eca15740b6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:24.023119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b00619b-d385-4ea6-a3b8-79eca15740b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:24.042497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-3b00619b-d385-4ea6-a3b8-79eca15740b6 None None] MFA Rules not processed for user `6fc9485505c443acb946ec66e0018f7b`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:24.043716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3b00619b-d385-4ea6-a3b8-79eca15740b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:24.050791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1222/2437] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:00:34.037509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:45.706383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:45.715879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e468a960-84d9-412b-97bd-93ab453b04b0 service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:45.720102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:45.720696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:45.720975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:45.721244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:45.722973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:45.790913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:45.792712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:45.806607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1216/2438] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:00:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:00:49.455604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b00619b-d385-4ea6-a3b8-79eca15740b6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:49.467643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b00619b-d385-4ea6-a3b8-79eca15740b6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:49.472996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:49.474176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:49.474459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:49.474733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:49.476495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:49.490068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:49.492206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:49.506539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1223/2439] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:00:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:00:49.520269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:49.522130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:49.535988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e786d02-8a87-4b9e-8004-ab1f1ef2d711 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:49.540858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:49.541477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:49.542057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:49.542415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] PATH_INFO: `/v3/users/7eb6ab0644bf4fa59d7bba232d6ccb3c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:49.543615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] RBAC: Authorizing `identity:delete_user(user_id=7eb6ab0644bf4fa59d7bba232d6ccb3c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:49.546114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:49.592309 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7eb6ab0644bf4fa59d7bba232d6ccb3c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.593037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7eb6ab0644bf4fa59d7bba232d6ccb3c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.600081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7eb6ab0644bf4fa59d7bba232d6ccb3c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.600716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7eb6ab0644bf4fa59d7bba232d6ccb3c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.612888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fd11217e-efff-403b-bfe3-f04f371b402a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7eb6ab0644bf4fa59d7bba232d6ccb3c'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.626175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1217/2440] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:00:49 2022] DELETE /identity/v3/users/7eb6ab0644bf4fa59d7bba232d6ccb3c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:49.794096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:49.818307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d1fc5-a105-41d6-ba52-f6354c76a792 service placement] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:49.878983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:49.879623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:49.879904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:49.880231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] PATH_INFO: `/v3/projects/57aaf234892543698d91303961f14abd` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:49.881336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57aaf234892543698d91303961f14abd)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:49.883234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:49.932138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57aaf234892543698d91303961f14abd', 'request_id': 'req-66c9284e-039e-4e22-b1fd-c9f5a7d50493'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.938675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57aaf234892543698d91303961f14abd', 'request_id': 'req-66c9284e-039e-4e22-b1fd-c9f5a7d50493'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.944214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57aaf234892543698d91303961f14abd', 'request_id': 'req-66c9284e-039e-4e22-b1fd-c9f5a7d50493'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.957050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] The token cache is being invalidate because project 57aaf234892543698d91303961f14abd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:00:49.957338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.958072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-66c9284e-039e-4e22-b1fd-c9f5a7d50493 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:49.959561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1224/2441] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:00:49 2022] DELETE /identity/v3/projects/57aaf234892543698d91303961f14abd => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:49.964619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780bcfc1-5f9a-43da-beb3-a97827fe4817 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:49.964911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780bcfc1-5f9a-43da-beb3-a97827fe4817 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:49.965184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780bcfc1-5f9a-43da-beb3-a97827fe4817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:49.986339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-780bcfc1-5f9a-43da-beb3-a97827fe4817 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:49.987547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-780bcfc1-5f9a-43da-beb3-a97827fe4817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:50.047088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1218/2442] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:50.056160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe8cc6-7e5e-484e-9947-eb8dd8e067cf None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.056454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe8cc6-7e5e-484e-9947-eb8dd8e067cf None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.056730 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfe8cc6-7e5e-484e-9947-eb8dd8e067cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.076679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4cfe8cc6-7e5e-484e-9947-eb8dd8e067cf None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:50.077874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4cfe8cc6-7e5e-484e-9947-eb8dd8e067cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:50.085613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1225/2443] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:50.089843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780bcfc1-5f9a-43da-beb3-a97827fe4817 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.100259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780bcfc1-5f9a-43da-beb3-a97827fe4817 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.104790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.105374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.105647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.105913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.107073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:00:50.107620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.109244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.110651 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:50.125694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1219/2444] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:00:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:00:50.126735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe8cc6-7e5e-484e-9947-eb8dd8e067cf None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.136125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfe8cc6-7e5e-484e-9947-eb8dd8e067cf None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.140923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.141508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.141787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.142055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.142681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.144595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.172131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1226/2445] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:00:50 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:00:50.176282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.186701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87cce83-990e-4424-90bf-e5cebc31cc33 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.241007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.241595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.241872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.242139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.242758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.244697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.266182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1220/2446] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:00:50 2022] POST /identity/v3/users => generated 293 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:00:50.270395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.280795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe95985a-cc83-4381-9450-3c531383beba admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.285507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.286097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.286379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.286649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.287281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.288919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.299301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1227/2447] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:50.303100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.313418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39c7b9d-939a-457e-a804-f969d26c67e9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.318083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a6bd15-7bf2-434d-be62-bb252d25f6dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.318676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a6bd15-7bf2-434d-be62-bb252d25f6dd admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.318940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a6bd15-7bf2-434d-be62-bb252d25f6dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.319208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a6bd15-7bf2-434d-be62-bb252d25f6dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.319836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a6bd15-7bf2-434d-be62-bb252d25f6dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.321458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a6bd15-7bf2-434d-be62-bb252d25f6dd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.331811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1221/2448] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:50.335687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.345392 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22b03af-29a8-4bd6-949a-5d4ed7d09ee3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.349833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.350456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.350732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.351001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] PATH_INFO: `/v3/projects/a4a4208580044b7eaf511cc235ccdb89/users/f736378bd65b456a9ec0196c194626e8/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.366480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4a4208580044b7eaf511cc235ccdb89, user_id=f736378bd65b456a9ec0196c194626e8, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.369123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.383163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1228/2449] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:00:50 2022] PUT /identity/v3/projects/a4a4208580044b7eaf511cc235ccdb89/users/f736378bd65b456a9ec0196c194626e8/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:50.394038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b3f43-76f8-4276-9d0c-35ad919c25a6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.394331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b3f43-76f8-4276-9d0c-35ad919c25a6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.394623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555b3f43-76f8-4276-9d0c-35ad919c25a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.422798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-555b3f43-76f8-4276-9d0c-35ad919c25a6 None None] MFA Rules not processed for user `f736378bd65b456a9ec0196c194626e8`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:50.424020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-555b3f43-76f8-4276-9d0c-35ad919c25a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:50.487429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1222/2450] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:50 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:50.491897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.505920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:50.556043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37c48ad-7935-4c12-bb9d-7b13e5936508 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.561507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.562128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.562407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.562678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.563332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.565329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.593438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1229/2451] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:00:50 2022] POST /identity/v3/projects => generated 385 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:00:50.597288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b3f43-76f8-4276-9d0c-35ad919c25a6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.607869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555b3f43-76f8-4276-9d0c-35ad919c25a6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.663189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.663781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.664093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.664360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.664967 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.666890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.675861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:50.696008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1223/2452] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:00:50 2022] POST /identity/v3/users => generated 299 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:00:50.697889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.709616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a2ec77-78c1-4029-b367-cd04eb5a266a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.714242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.714835 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.715113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.715382 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.716052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.717645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.728385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1230/2453] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:00:50.732798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.742784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f724f126-5c86-47c6-a283-cb4c74c5c0ee admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.747965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.748703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.748984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.749254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] PATH_INFO: `/v3/projects/0a084df7a9e64f58a895342ad8cabd49/users/839240da8e114bef9ac7f70b4c80881a/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.764619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a084df7a9e64f58a895342ad8cabd49, user_id=839240da8e114bef9ac7f70b4c80881a, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.767258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.779879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1224/2454] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:00:50 2022] PUT /identity/v3/projects/0a084df7a9e64f58a895342ad8cabd49/users/839240da8e114bef9ac7f70b4c80881a/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:50.783753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.793481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d1f1dc-d84c-4e86-a2b0-5e95f2fc9319 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.849657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8456aee2-9f73-4287-8a62-8530ca4a8901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.850256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8456aee2-9f73-4287-8a62-8530ca4a8901 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.850531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8456aee2-9f73-4287-8a62-8530ca4a8901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.850796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8456aee2-9f73-4287-8a62-8530ca4a8901 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.851434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8456aee2-9f73-4287-8a62-8530ca4a8901 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.853030 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8456aee2-9f73-4287-8a62-8530ca4a8901 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.862865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1231/2455] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:50 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:50.867032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:50.878304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364f5a55-f432-41dd-9196-203f58c5700a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:50.882949 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:50.883704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.884054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.884355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] PATH_INFO: `/v3/projects/0a084df7a9e64f58a895342ad8cabd49/users/839240da8e114bef9ac7f70b4c80881a/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.886219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a084df7a9e64f58a895342ad8cabd49, user_id=839240da8e114bef9ac7f70b4c80881a, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:50.889125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:50.899009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1225/2456] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:00:50 2022] PUT /identity/v3/projects/0a084df7a9e64f58a895342ad8cabd49/users/839240da8e114bef9ac7f70b4c80881a/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:50.908961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884546e0-89f1-46c5-b58b-b9dd51dc955c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:50.909248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884546e0-89f1-46c5-b58b-b9dd51dc955c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:50.909519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884546e0-89f1-46c5-b58b-b9dd51dc955c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:50.938072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-884546e0-89f1-46c5-b58b-b9dd51dc955c None None] MFA Rules not processed for user `839240da8e114bef9ac7f70b4c80881a`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:50.939262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-884546e0-89f1-46c5-b58b-b9dd51dc955c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:51.006473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1232/2457] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:50 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:51.017901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:51.019692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:51.078752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243ec401-aaf0-41f3-ae26-9327192ed51a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:51.082506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b674c282-76e5-448c-923f-ccc7a2d7f661 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:51.083196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b674c282-76e5-448c-923f-ccc7a2d7f661 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:51.083602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b674c282-76e5-448c-923f-ccc7a2d7f661 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:51.083879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b674c282-76e5-448c-923f-ccc7a2d7f661 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:51.094491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b674c282-76e5-448c-923f-ccc7a2d7f661 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:51.095958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b674c282-76e5-448c-923f-ccc7a2d7f661 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:51.107158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1226/2458] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:00:51 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:00:51.174381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884546e0-89f1-46c5-b58b-b9dd51dc955c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:51.184440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:51.238875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884546e0-89f1-46c5-b58b-b9dd51dc955c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:51.243301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:51.243924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:51.244240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:51.244512 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] PATH_INFO: `/v3/users/f736378bd65b456a9ec0196c194626e8` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:51.245639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f736378bd65b456a9ec0196c194626e8)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:51.247874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:51.293399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f736378bd65b456a9ec0196c194626e8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.294635 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f736378bd65b456a9ec0196c194626e8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.295439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f736378bd65b456a9ec0196c194626e8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.302413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f736378bd65b456a9ec0196c194626e8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.314247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f736378bd65b456a9ec0196c194626e8'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.327500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1233/2459] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:00:51 2022] DELETE /identity/v3/users/f736378bd65b456a9ec0196c194626e8 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:51.332045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:51.332330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:51.332612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:51.351359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:51.352562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:51.412196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1227/2460] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:51.421960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:51.423343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:51.482087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b5dcc6-3211-4774-b47a-0c3ff7e7ca27 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:51.485827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:51.486423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:51.486709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:51.486978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:51.496227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:51.498256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:51.508295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1234/2461] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:00:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:00:51.774992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:51.776580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:51.791308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0652c42-c30f-4e45-966a-2f1d161022b9 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:51.808782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:51.808782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:51.808782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:51.808782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] PATH_INFO: `/v3/projects/a4a4208580044b7eaf511cc235ccdb89` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:51.809217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4a4208580044b7eaf511cc235ccdb89)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:51.810052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:51.856735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4a4208580044b7eaf511cc235ccdb89', 'request_id': 'req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.858709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=4,keystone:SELECT=92,keystone:DELETE=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:51.863359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4a4208580044b7eaf511cc235ccdb89', 'request_id': 'req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.869894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4a4208580044b7eaf511cc235ccdb89', 'request_id': 'req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.882163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] The token cache is being invalidate because project a4a4208580044b7eaf511cc235ccdb89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:00:51.882448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.883058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:51.884563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1228/2462] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:00:51 2022] DELETE /identity/v3/projects/a4a4208580044b7eaf511cc235ccdb89 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:00:51.889311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:51.898997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6cd7f-76d7-40af-9400-53c159813aab service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:51.928129 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:51.958308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:51.958934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:51.959217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:51.959484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] PATH_INFO: `/v3/users/839240da8e114bef9ac7f70b4c80881a` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:51.960604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] RBAC: Authorizing `identity:delete_user(user_id=839240da8e114bef9ac7f70b4c80881a)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:51.962535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.010602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '839240da8e114bef9ac7f70b4c80881a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.011247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '839240da8e114bef9ac7f70b4c80881a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.011849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '839240da8e114bef9ac7f70b4c80881a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.019577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '839240da8e114bef9ac7f70b4c80881a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.031286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ca6bb5a5-042f-48b6-a804-a211e01f7890 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '839240da8e114bef9ac7f70b4c80881a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.044092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1235/2463] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:00:51 2022] DELETE /identity/v3/users/839240da8e114bef9ac7f70b4c80881a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:52.303535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:52.304968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:52.319767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc689ff-ceb3-427f-96ba-b8ff8e8d69c9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:52.377694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:52.378315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.378591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.378858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] PATH_INFO: `/v3/projects/0a084df7a9e64f58a895342ad8cabd49` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.379984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a084df7a9e64f58a895342ad8cabd49)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:52.381910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.427166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a084df7a9e64f58a895342ad8cabd49', 'request_id': 'req-5eaf7c23-f596-40fd-b511-09036fb1cb23'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.433509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a084df7a9e64f58a895342ad8cabd49', 'request_id': 'req-5eaf7c23-f596-40fd-b511-09036fb1cb23'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.439113 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a084df7a9e64f58a895342ad8cabd49', 'request_id': 'req-5eaf7c23-f596-40fd-b511-09036fb1cb23'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.451514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] The token cache is being invalidate because project 0a084df7a9e64f58a895342ad8cabd49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:00:52.451793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.452748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5eaf7c23-f596-40fd-b511-09036fb1cb23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:00:52.454207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1229/2464] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:00:52 2022] DELETE /identity/v3/projects/0a084df7a9e64f58a895342ad8cabd49 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:52.459062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66cf32-a40d-4e51-ae42-124aff5bd7a6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.459452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66cf32-a40d-4e51-ae42-124aff5bd7a6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.459733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b66cf32-a40d-4e51-ae42-124aff5bd7a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.480476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-1b66cf32-a40d-4e51-ae42-124aff5bd7a6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:52.481697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1b66cf32-a40d-4e51-ae42-124aff5bd7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:52.542741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1236/2465] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:52.551877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da71485-91c8-4a42-93ed-3fe0fa9615dd None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.552128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da71485-91c8-4a42-93ed-3fe0fa9615dd None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.552404 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da71485-91c8-4a42-93ed-3fe0fa9615dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.571733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5da71485-91c8-4a42-93ed-3fe0fa9615dd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:52.572955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5da71485-91c8-4a42-93ed-3fe0fa9615dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:52.581443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1230/2466] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:52.585624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66cf32-a40d-4e51-ae42-124aff5bd7a6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:52.596904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b66cf32-a40d-4e51-ae42-124aff5bd7a6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:52.601675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:52.602274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.602554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.602822 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.603469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:00:52.603765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:52.605594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.619158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1237/2467] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:00:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:00:52.624261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da71485-91c8-4a42-93ed-3fe0fa9615dd None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:52.633884 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da71485-91c8-4a42-93ed-3fe0fa9615dd None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:52.638711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:52.639296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.639577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.639847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.640515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:52.642419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.668605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1231/2468] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:00:52 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:00:52.673616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:52.683504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee916cf1-73c2-48e7-9deb-d3491f2d5fc1 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:52.717795 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:52.742238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:52.742832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.743111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.743380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.744036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:52.745948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.768040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1238/2469] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:00:52 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:00:52.772260 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:52.782412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db89438-af6a-473a-a9de-64035a87824a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:52.787176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:52.787778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.788098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.788400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.789010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:52.790605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.800985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1232/2470] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:52.804966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:52.815682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e62be-2cd1-4a66-89d1-9a711d847109 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:52.820306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941a66b2-9a7f-4170-a9db-e4fa040be898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:52.820896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a66b2-9a7f-4170-a9db-e4fa040be898 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.821173 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a66b2-9a7f-4170-a9db-e4fa040be898 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.821443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941a66b2-9a7f-4170-a9db-e4fa040be898 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.822079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a66b2-9a7f-4170-a9db-e4fa040be898 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:52.823659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941a66b2-9a7f-4170-a9db-e4fa040be898 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.834040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1239/2471] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:52 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:52.838126 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:52.848145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62c47d0-7409-45bb-85be-2393a3844e0e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:52.852775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:52.853430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.853708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.853975 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] PATH_INFO: `/v3/projects/6df5f1d061e546589047c20010d79a81/users/33f136b11c3b4b4dbd6e9b59b5cc65dd/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.869008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6df5f1d061e546589047c20010d79a81, user_id=33f136b11c3b4b4dbd6e9b59b5cc65dd, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:52.871687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:52.885222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1233/2472] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:00:52 2022] PUT /identity/v3/projects/6df5f1d061e546589047c20010d79a81/users/33f136b11c3b4b4dbd6e9b59b5cc65dd/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:52.895990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a611145-f060-4e58-802d-74ac0a49b210 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:52.896285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a611145-f060-4e58-802d-74ac0a49b210 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:52.896595 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a611145-f060-4e58-802d-74ac0a49b210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:52.925238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6a611145-f060-4e58-802d-74ac0a49b210 None None] MFA Rules not processed for user `33f136b11c3b4b4dbd6e9b59b5cc65dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:52.926445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6a611145-f060-4e58-802d-74ac0a49b210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:52.989483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1240/2473] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:52.993960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.003919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:53.060873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a07efe-3b56-40c9-90f1-0cae2772a4e6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.065384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.065972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.066247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.066519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.067143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.069069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.097062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1234/2474] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:00:52 2022] POST /identity/v3/projects => generated 385 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:00:53.101113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a611145-f060-4e58-802d-74ac0a49b210 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.112602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a611145-f060-4e58-802d-74ac0a49b210 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.168108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.168716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.169647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.169917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.170541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.172492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.194486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1241/2475] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:00:53 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:00:53.198049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.208662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d14562-d3e3-46ed-815c-381d08069595 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.213604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.214222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.214534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.214807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.215439 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.217380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.227633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1235/2476] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:53.232038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.241933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f917e6f9-16c1-4f2e-abd7-23019d041946 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.246606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8ae72c-4eeb-4817-94f7-3b3c25b37514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.247895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ae72c-4eeb-4817-94f7-3b3c25b37514 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.248274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ae72c-4eeb-4817-94f7-3b3c25b37514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.248545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8ae72c-4eeb-4817-94f7-3b3c25b37514 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.249180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8ae72c-4eeb-4817-94f7-3b3c25b37514 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.250756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8ae72c-4eeb-4817-94f7-3b3c25b37514 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.261340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1242/2477] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:53.265407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.275130 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfd0478-6b6f-42d2-9e28-67b31a5bbd45 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.279808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.280506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.280829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.281025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] PATH_INFO: `/v3/projects/b88423e66f084b9695627b38083bb7e5/users/fc8d0002301b4ae2b462d76333859b02/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.296714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b88423e66f084b9695627b38083bb7e5, user_id=fc8d0002301b4ae2b462d76333859b02, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.299350 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.312414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1236/2478] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:00:53 2022] PUT /identity/v3/projects/b88423e66f084b9695627b38083bb7e5/users/fc8d0002301b4ae2b462d76333859b02/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:00:53.322540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c37ff6-4fc0-4dda-ab43-23d961a9753b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.322829 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c37ff6-4fc0-4dda-ab43-23d961a9753b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.323125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c37ff6-4fc0-4dda-ab43-23d961a9753b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.350649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b3c37ff6-4fc0-4dda-ab43-23d961a9753b None None] MFA Rules not processed for user `fc8d0002301b4ae2b462d76333859b02`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:53.351834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b3c37ff6-4fc0-4dda-ab43-23d961a9753b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:53.364140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:53.418539 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1243/2479] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:53.423572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.485481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c75fa0-9614-4959-a244-82ea8693eb29 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.490172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.490755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.491031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.491294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.491904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.493816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.519657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1237/2480] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:00:53 2022] POST /identity/v3/projects => generated 387 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:00:53.523778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c37ff6-4fc0-4dda-ab43-23d961a9753b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.534974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c37ff6-4fc0-4dda-ab43-23d961a9753b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.590748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.591634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.591634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.591849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.592511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.594419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.615454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1244/2481] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:00:53 2022] POST /identity/v3/users => generated 300 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:00:53.619376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.630293 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535149b8-99d0-4c27-a55e-96833487f735 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.634881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.635462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.635735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.636036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.636666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.638934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.649125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1238/2482] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:53.653065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.662897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01bd57a-f1e1-49bd-a84a-ff77559233ac admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.667573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.668238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.668519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.668788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] PATH_INFO: `/v3/projects/8059a0a09bb841798936b90bb5b1635b/users/929015ece5994c578ee9941156dff7c4/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.683353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8059a0a09bb841798936b90bb5b1635b, user_id=929015ece5994c578ee9941156dff7c4, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.686874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.699484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1245/2483] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:00:53 2022] PUT /identity/v3/projects/8059a0a09bb841798936b90bb5b1635b/users/929015ece5994c578ee9941156dff7c4/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:53.704146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.714207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5944e7-30f5-42c6-af78-1147a6be17ec admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.742635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:53.772292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb5ff16-a971-4cb6-8c72-73cf44489315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.772977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb5ff16-a971-4cb6-8c72-73cf44489315 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.773255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb5ff16-a971-4cb6-8c72-73cf44489315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.773545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb5ff16-a971-4cb6-8c72-73cf44489315 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.774168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb5ff16-a971-4cb6-8c72-73cf44489315 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.775766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb5ff16-a971-4cb6-8c72-73cf44489315 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.785442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1239/2484] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:00:53 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:00:53.789998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.801425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc05862-b8dc-4c28-a0f2-7de7085f8e50 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:53.806035 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:53.806677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.806951 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.807235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] PATH_INFO: `/v3/projects/8059a0a09bb841798936b90bb5b1635b/users/929015ece5994c578ee9941156dff7c4/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.809159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8059a0a09bb841798936b90bb5b1635b, user_id=929015ece5994c578ee9941156dff7c4, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:53.812804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:53.823063 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1246/2485] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:00:53 2022] PUT /identity/v3/projects/8059a0a09bb841798936b90bb5b1635b/users/929015ece5994c578ee9941156dff7c4/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:00:53.834440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:53.834730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:53.835000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:53.863013 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] MFA Rules not processed for user `929015ece5994c578ee9941156dff7c4`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:00:53.864270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:53.931367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1240/2486] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:00:53 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:00:53.942686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:53.944423 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:54.001229 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db82d1-86f9-452f-8055-8d7732893356 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:54.005151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:54.005737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:54.006011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:54.006273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:54.016263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:54.017669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:54.026974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1247/2487] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:00:53 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:00:54.520240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:54.521628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:00:54.533818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0579a2c-f8db-4302-a3a1-2995dddad077 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:54.537352 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:54.537929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:54.538201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:54.538465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:54.594523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:54.595963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:54.604600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1241/2488] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:00:54 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:00:58.636851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:00:58.646283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba7ee6f-3489-4633-8c63-5e7b66842f5a service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:00:58.649973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012300a0-e1ac-4203-9003-0515a139f23f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:00:58.650553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012300a0-e1ac-4203-9003-0515a139f23f service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:00:58.650826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012300a0-e1ac-4203-9003-0515a139f23f service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:00:58.651089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012300a0-e1ac-4203-9003-0515a139f23f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:00:58.657912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:00:58.711210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012300a0-e1ac-4203-9003-0515a139f23f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:00:58.712668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012300a0-e1ac-4203-9003-0515a139f23f service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:00:58.721547 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1248/2489] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:00:58 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:00.812156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:00.862736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:00.878158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abca88ce-2b38-4237-9465-31d422116b1a service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:00.882653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:00.883258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:00.883534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:00.883798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] PATH_INFO: `/v3/users/33f136b11c3b4b4dbd6e9b59b5cc65dd` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:00.884923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] RBAC: Authorizing `identity:delete_user(user_id=33f136b11c3b4b4dbd6e9b59b5cc65dd)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:00.886819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:00.931756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33f136b11c3b4b4dbd6e9b59b5cc65dd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:00.932431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33f136b11c3b4b4dbd6e9b59b5cc65dd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:00.940114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33f136b11c3b4b4dbd6e9b59b5cc65dd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:00.940737 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33f136b11c3b4b4dbd6e9b59b5cc65dd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:00.953044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33f136b11c3b4b4dbd6e9b59b5cc65dd'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:00.965399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1242/2490] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:01:00 2022] DELETE /identity/v3/users/33f136b11c3b4b4dbd6e9b59b5cc65dd => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:00.969931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:00.970780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:00.971050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:00.990907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:00.992135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:01.058096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1249/2491] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:01.068066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:01.069666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:01.127085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63320860-7081-4eaf-b330-e50e70e0d39c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:01.130684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:01.131274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:01.131551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:01.131823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:01.141490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:01.142906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:01.152119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1243/2492] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:01.447805 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:01.449610 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:01.469115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908d9a7a-f19e-4935-83ea-5dffc2fb68fa None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:01.474838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:01.476413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:01.476700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:01.476973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] PATH_INFO: `/v3/projects/6df5f1d061e546589047c20010d79a81` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:01.478577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6df5f1d061e546589047c20010d79a81)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:01.483268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:01.535380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6df5f1d061e546589047c20010d79a81', 'request_id': 'req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.541961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6df5f1d061e546589047c20010d79a81', 'request_id': 'req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.549292 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6df5f1d061e546589047c20010d79a81', 'request_id': 'req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.562080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] The token cache is being invalidate because project 6df5f1d061e546589047c20010d79a81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:01:01.562307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.562941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.564478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1250/2493] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:01:01 2022] DELETE /identity/v3/projects/6df5f1d061e546589047c20010d79a81 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:01:01.568567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:01.581085 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5df3b9e-066b-47fc-b73d-9c71f8513775 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:01.639402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:01.640081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:01.640397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:01.640630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] PATH_INFO: `/v3/users/fc8d0002301b4ae2b462d76333859b02` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:01.642101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc8d0002301b4ae2b462d76333859b02)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:01.643729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:01.657693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:01.691581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc8d0002301b4ae2b462d76333859b02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.692345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc8d0002301b4ae2b462d76333859b02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.699455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc8d0002301b4ae2b462d76333859b02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.700592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc8d0002301b4ae2b462d76333859b02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.713027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc8d0002301b4ae2b462d76333859b02'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:01.726863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1244/2494] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:01:01 2022] DELETE /identity/v3/users/fc8d0002301b4ae2b462d76333859b02 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:02.007444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:02.009142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:02.025424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8de54f2-7f45-4bfd-a477-f8e51b3f1199 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:02.060022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:02.094534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:02.095187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:02.095438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:02.095710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] PATH_INFO: `/v3/projects/b88423e66f084b9695627b38083bb7e5` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:02.096885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b88423e66f084b9695627b38083bb7e5)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:02.098793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:02.146214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b88423e66f084b9695627b38083bb7e5', 'request_id': 'req-960199f4-2171-44ce-9f8b-8d66c5b05951'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.152430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b88423e66f084b9695627b38083bb7e5', 'request_id': 'req-960199f4-2171-44ce-9f8b-8d66c5b05951'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.158893 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b88423e66f084b9695627b38083bb7e5', 'request_id': 'req-960199f4-2171-44ce-9f8b-8d66c5b05951'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.171758 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] The token cache is being invalidate because project b88423e66f084b9695627b38083bb7e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:01:02.172109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.172751 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.174341 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1251/2495] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:01:02 2022] DELETE /identity/v3/projects/b88423e66f084b9695627b38083bb7e5 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:02.178373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:02.179789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:02.193182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2e11c-ffc7-4eb0-b2f3-b48b4d7af6d1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:02.250158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:02.250777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:02.251056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:02.251325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] PATH_INFO: `/v3/users/929015ece5994c578ee9941156dff7c4` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:02.252705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] RBAC: Authorizing `identity:delete_user(user_id=929015ece5994c578ee9941156dff7c4)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:02.254639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:02.299394 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '929015ece5994c578ee9941156dff7c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.300062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '929015ece5994c578ee9941156dff7c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.306798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '929015ece5994c578ee9941156dff7c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.307425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '929015ece5994c578ee9941156dff7c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.319117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-db781428-3405-4d17-a55f-ef9b8c6d958e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '929015ece5994c578ee9941156dff7c4'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.332017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1245/2496] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:01:02 2022] DELETE /identity/v3/users/929015ece5994c578ee9941156dff7c4 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:02.600800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:02.602340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:02.617273 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-960199f4-2171-44ce-9f8b-8d66c5b05951 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:02.676597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:02.677228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:02.677504 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:02.677774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] PATH_INFO: `/v3/projects/8059a0a09bb841798936b90bb5b1635b` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:02.678876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8059a0a09bb841798936b90bb5b1635b)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:02.680797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:02.726728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8059a0a09bb841798936b90bb5b1635b', 'request_id': 'req-e011b95c-ea43-4989-b5d2-562fca2d90f2'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.733299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8059a0a09bb841798936b90bb5b1635b', 'request_id': 'req-e011b95c-ea43-4989-b5d2-562fca2d90f2'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.738861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8059a0a09bb841798936b90bb5b1635b', 'request_id': 'req-e011b95c-ea43-4989-b5d2-562fca2d90f2'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.740520 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:02.752970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] The token cache is being invalidate because project 8059a0a09bb841798936b90bb5b1635b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:01:02.753262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.753909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e011b95c-ea43-4989-b5d2-562fca2d90f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:02.755375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1252/2497] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:01:02 2022] DELETE /identity/v3/projects/8059a0a09bb841798936b90bb5b1635b => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:02.761245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05faf85-3510-40e4-a873-5f8fd68cbff3 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:02.761544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05faf85-3510-40e4-a873-5f8fd68cbff3 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:02.761813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d05faf85-3510-40e4-a873-5f8fd68cbff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:02.781798 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d05faf85-3510-40e4-a873-5f8fd68cbff3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:02.783033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d05faf85-3510-40e4-a873-5f8fd68cbff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:02.788363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:02.845119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1246/2498] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:02.853848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b22ea-408b-4a32-a337-7b6f1b499b1f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:02.854168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b22ea-408b-4a32-a337-7b6f1b499b1f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:02.854443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90b22ea-408b-4a32-a337-7b6f1b499b1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:02.874489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b90b22ea-408b-4a32-a337-7b6f1b499b1f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:02.875676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b90b22ea-408b-4a32-a337-7b6f1b499b1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:02.884879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1253/2499] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:02.889617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05faf85-3510-40e4-a873-5f8fd68cbff3 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:02.899289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d05faf85-3510-40e4-a873-5f8fd68cbff3 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:02.904481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:02.905067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:02.905343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:02.905611 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:02.906241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:01:02.906537 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:02.908146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:02.919387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1247/2500] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:01:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:01:02.923649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b22ea-408b-4a32-a337-7b6f1b499b1f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:02.932760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90b22ea-408b-4a32-a337-7b6f1b499b1f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:02.937577 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:02.938159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:02.938433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:02.938698 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:02.939311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:02.941221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:02.969701 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1254/2501] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:01:02 2022] POST /identity/v3/projects => generated 381 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:01:02.973846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:02.983994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7784790-333a-4dc2-b038-9076173cf9b6 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.039756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.040371 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.040652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.040922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.041539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.043551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.064367 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1248/2502] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:01:02 2022] POST /identity/v3/users => generated 291 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:01:03.068665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.079858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd61f9b6-f6ef-4262-a6dc-e4b38e228cdc admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.084541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.085132 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.085411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.085681 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.086313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.087892 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.098889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1255/2503] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:03.102735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.112818 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328aa64a-d8be-4fc2-b9a7-9ae9ff745343 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.117380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecdd21ff-1a8f-4172-9487-9c42d91622a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.117961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdd21ff-1a8f-4172-9487-9c42d91622a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.118240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdd21ff-1a8f-4172-9487-9c42d91622a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.118510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecdd21ff-1a8f-4172-9487-9c42d91622a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.119140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdd21ff-1a8f-4172-9487-9c42d91622a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.120767 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecdd21ff-1a8f-4172-9487-9c42d91622a3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.131469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1249/2504] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:03.135442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.145957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46741c81-4a48-415c-a517-324ce6915b9e admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.150756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.151397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.151679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.151980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] PATH_INFO: `/v3/projects/4c6a714b8c484037bf68c1e76f4df030/users/d22f3623d1b141658518ca66fd60a9d6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.166994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c6a714b8c484037bf68c1e76f4df030, user_id=d22f3623d1b141658518ca66fd60a9d6, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.169932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.183350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1256/2505] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:01:03 2022] PUT /identity/v3/projects/4c6a714b8c484037bf68c1e76f4df030/users/d22f3623d1b141658518ca66fd60a9d6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:03.195897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092ecc8-1aa2-45f8-9832-d1e222cc3be6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.196423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092ecc8-1aa2-45f8-9832-d1e222cc3be6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.196542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092ecc8-1aa2-45f8-9832-d1e222cc3be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.224872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5092ecc8-1aa2-45f8-9832-d1e222cc3be6 None None] MFA Rules not processed for user `d22f3623d1b141658518ca66fd60a9d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:03.226088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5092ecc8-1aa2-45f8-9832-d1e222cc3be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:03.290483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1250/2506] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:03 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:03.295197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.356674 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c974eab-52ba-40a7-a7f4-536642136500 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.361329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.361960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.362248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.362557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.363277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.365301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.395804 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1257/2507] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:01:03 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:01:03.396973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092ecc8-1aa2-45f8-9832-d1e222cc3be6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.407770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092ecc8-1aa2-45f8-9832-d1e222cc3be6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.411388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:03.466543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.467142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.467419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.467685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.468349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.470259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.491650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1251/2508] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:01:03 2022] POST /identity/v3/users => generated 298 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:01:03.497027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.507537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655b512e-4d11-4815-a6bb-f3292ca559a3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.513021 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.513611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.513887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.514152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.514778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.516401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.528351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1258/2509] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:03.532460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.542395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8993ec77-6d61-4991-8a7c-f74c5529e2d5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.547593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.548256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.548532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.548803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] PATH_INFO: `/v3/projects/1a676e96cc1548d5a00836b62f31864b/users/94c69a252688401a8c3c11d8440097d6/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.564200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a676e96cc1548d5a00836b62f31864b, user_id=94c69a252688401a8c3c11d8440097d6, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.566867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.579590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1252/2510] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:01:03 2022] PUT /identity/v3/projects/1a676e96cc1548d5a00836b62f31864b/users/94c69a252688401a8c3c11d8440097d6/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:03.583391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.593573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc98def-6e1e-4cf1-a5f5-060374f7fbbd admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.649219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643263c2-4d78-439f-bc01-c9799912da25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.649719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643263c2-4d78-439f-bc01-c9799912da25 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.649996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643263c2-4d78-439f-bc01-c9799912da25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.650265 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643263c2-4d78-439f-bc01-c9799912da25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.650891 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643263c2-4d78-439f-bc01-c9799912da25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.652517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643263c2-4d78-439f-bc01-c9799912da25 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.655331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:03.664936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1259/2511] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:03.669323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.680123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac43e73b-bb2d-4bc3-ac1d-6adb10b3fbf0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.685025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.685659 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.685940 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.686207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] PATH_INFO: `/v3/projects/1a676e96cc1548d5a00836b62f31864b/users/94c69a252688401a8c3c11d8440097d6/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.688145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a676e96cc1548d5a00836b62f31864b, user_id=94c69a252688401a8c3c11d8440097d6, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.690755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.701385 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1253/2512] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:01:03 2022] PUT /identity/v3/projects/1a676e96cc1548d5a00836b62f31864b/users/94c69a252688401a8c3c11d8440097d6/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:03.711375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8160e2-c2e1-471f-b660-cad73d66847f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.711667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8160e2-c2e1-471f-b660-cad73d66847f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.711963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8160e2-c2e1-471f-b660-cad73d66847f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.740636 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4b8160e2-c2e1-471f-b660-cad73d66847f None None] MFA Rules not processed for user `94c69a252688401a8c3c11d8440097d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:03.741855 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4b8160e2-c2e1-471f-b660-cad73d66847f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:03.814617 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1260/2513] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:03 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:03.827222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:03.829141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:03.892043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90efaf9e-50d0-42c5-98b9-7960c62b6ba2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:03.895752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:03.896348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:03.896627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:03.896925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:03.907003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:03.908462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:03.918281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1254/2514] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:03 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:03.992062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8160e2-c2e1-471f-b660-cad73d66847f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:04.003708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8160e2-c2e1-471f-b660-cad73d66847f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:04.007247 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3c446c-5a6e-416b-a460-1c9674362348 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:04.007830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3c446c-5a6e-416b-a460-1c9674362348 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:04.008147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3c446c-5a6e-416b-a460-1c9674362348 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:04.008416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3c446c-5a6e-416b-a460-1c9674362348 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:04.066990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3c446c-5a6e-416b-a460-1c9674362348 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:04.068446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3c446c-5a6e-416b-a460-1c9674362348 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:04.077175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1261/2515] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:03 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:04.350052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:04.412312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde620bb-9dcd-4b9a-89c4-a15dfcb6818f service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:04.416686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:04.417297 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:04.417576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:04.417846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] PATH_INFO: `/v3/users/d22f3623d1b141658518ca66fd60a9d6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:04.419031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d22f3623d1b141658518ca66fd60a9d6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:04.420980 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:04.423254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:04.468740 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd22f3623d1b141658518ca66fd60a9d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:04.469375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd22f3623d1b141658518ca66fd60a9d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:04.476113 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd22f3623d1b141658518ca66fd60a9d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:04.476727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd22f3623d1b141658518ca66fd60a9d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:04.489483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd22f3623d1b141658518ca66fd60a9d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:04.501609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1255/2516] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:01:04 2022] DELETE /identity/v3/users/d22f3623d1b141658518ca66fd60a9d6 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:01:04.506451 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:04.506852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:04.507226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:04.529766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:04.530948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:04.588389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1262/2517] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:04.598987 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:04.600415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:04.659016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acf8440-88eb-4af7-a599-589f242632d5 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:04.662658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:04.663187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:04.663459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:04.663723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:04.672962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:04.674362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:04.683633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1256/2518] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:04.957299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:04.958732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:04.970929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf834eda-dc2a-4bb3-b6ed-d6ccea319615 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:04.975825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:04.976494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:04.976798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:04.977037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] PATH_INFO: `/v3/projects/4c6a714b8c484037bf68c1e76f4df030` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:04.978234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c6a714b8c484037bf68c1e76f4df030)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:04.980073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:04.998043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:05.028405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c6a714b8c484037bf68c1e76f4df030', 'request_id': 'req-701da008-ac97-44c9-853f-26e690ea51ae'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.034645 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c6a714b8c484037bf68c1e76f4df030', 'request_id': 'req-701da008-ac97-44c9-853f-26e690ea51ae'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.040422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c6a714b8c484037bf68c1e76f4df030', 'request_id': 'req-701da008-ac97-44c9-853f-26e690ea51ae'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.053799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] The token cache is being invalidate because project 4c6a714b8c484037bf68c1e76f4df030 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:01:05.054086 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.054718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.056201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1263/2519] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:01:04 2022] DELETE /identity/v3/projects/4c6a714b8c484037bf68c1e76f4df030 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:05.060799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:05.070369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915cd7c3-d262-49b2-9f83-47fc3ff0ac97 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:05.125990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:05.126608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.127138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.127410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] PATH_INFO: `/v3/users/94c69a252688401a8c3c11d8440097d6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.128535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94c69a252688401a8c3c11d8440097d6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:05.130429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:05.157935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:05.177676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94c69a252688401a8c3c11d8440097d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.178308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94c69a252688401a8c3c11d8440097d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.185852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94c69a252688401a8c3c11d8440097d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.186482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94c69a252688401a8c3c11d8440097d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.198189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cbb0d1f4-1299-43e8-9b8b-027eac7fc384 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94c69a252688401a8c3c11d8440097d6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.211649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1257/2520] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:01:05 2022] DELETE /identity/v3/users/94c69a252688401a8c3c11d8440097d6 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:05.496393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:05.498235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:05.512596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701da008-ac97-44c9-853f-26e690ea51ae admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:05.571110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:05.571740 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.572056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.572329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] PATH_INFO: `/v3/projects/1a676e96cc1548d5a00836b62f31864b` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.573452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a676e96cc1548d5a00836b62f31864b)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:05.575472 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:05.622745 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a676e96cc1548d5a00836b62f31864b', 'request_id': 'req-8b2c2c05-b766-4b77-91db-4c99ae7958a1'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.629227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a676e96cc1548d5a00836b62f31864b', 'request_id': 'req-8b2c2c05-b766-4b77-91db-4c99ae7958a1'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.634852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a676e96cc1548d5a00836b62f31864b', 'request_id': 'req-8b2c2c05-b766-4b77-91db-4c99ae7958a1'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.646924 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] The token cache is being invalidate because project 1a676e96cc1548d5a00836b62f31864b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:01:05.647213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.647840 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-8b2c2c05-b766-4b77-91db-4c99ae7958a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:05.649347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1264/2521] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:01:05 2022] DELETE /identity/v3/projects/1a676e96cc1548d5a00836b62f31864b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:05.655697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb63e3e-efa0-4d13-ac3b-6f21df631e14 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.656186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb63e3e-efa0-4d13-ac3b-6f21df631e14 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.656279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb63e3e-efa0-4d13-ac3b-6f21df631e14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.676554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8eb63e3e-efa0-4d13-ac3b-6f21df631e14 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:05.677766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8eb63e3e-efa0-4d13-ac3b-6f21df631e14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:05.736544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1258/2522] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:05.744903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c98bbfc-f53a-4287-ae29-218dfb928267 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.745154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c98bbfc-f53a-4287-ae29-218dfb928267 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.745430 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c98bbfc-f53a-4287-ae29-218dfb928267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.765697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-3c98bbfc-f53a-4287-ae29-218dfb928267 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:05.766895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3c98bbfc-f53a-4287-ae29-218dfb928267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:05.774915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1265/2523] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:05.779598 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb63e3e-efa0-4d13-ac3b-6f21df631e14 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:05.789518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb63e3e-efa0-4d13-ac3b-6f21df631e14 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:05.794753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:05.795377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.795676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.795965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.796650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:01:05.796950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:05.798645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:05.812090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1259/2524] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:01:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:01:05.816411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c98bbfc-f53a-4287-ae29-218dfb928267 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:05.826494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c98bbfc-f53a-4287-ae29-218dfb928267 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:05.831078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:05.831657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.831957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.832224 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.832851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:05.834854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:05.863148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1266/2525] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:01:05 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:01:05.867346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:05.877510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d45edb1-0ba3-41a7-a881-fe46eec37084 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:05.934387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:05.935052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.935336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.935607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.936258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:05.938160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:05.963080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1260/2526] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:01:05 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:01:05.967973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:05.978665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0767ed-93c3-4f78-b5f7-bea30d0a48ac admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:05.984032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:05.984568 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:05.984906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:05.985169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:05.985802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:05.987411 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:05.988818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:05.999429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1267/2527] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:06.003414 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:06.014132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583404cb-ee07-4cfb-92b9-40960279576d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:06.018693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec42086-a180-4545-8058-787eb8d87dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:06.019281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec42086-a180-4545-8058-787eb8d87dd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.019555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec42086-a180-4545-8058-787eb8d87dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.019821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec42086-a180-4545-8058-787eb8d87dd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.020502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec42086-a180-4545-8058-787eb8d87dd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:06.022078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec42086-a180-4545-8058-787eb8d87dd6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:06.033761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1261/2528] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:06.037821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:06.047864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba2d5a4-3b80-4239-89ac-d065e816db74 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:06.054448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:06.055578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.056149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.056643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] PATH_INFO: `/v3/projects/5c5e62bd9fa245908de33d5f0a954670/users/38a31ab7e2fa4ca89ad081ab4871cd5b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.074398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c5e62bd9fa245908de33d5f0a954670, user_id=38a31ab7e2fa4ca89ad081ab4871cd5b, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:06.077687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:06.091148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1268/2529] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:01:06 2022] PUT /identity/v3/projects/5c5e62bd9fa245908de33d5f0a954670/users/38a31ab7e2fa4ca89ad081ab4871cd5b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:06.102242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254e09c-d621-40b4-a907-af7a67fe9872 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.102599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254e09c-d621-40b4-a907-af7a67fe9872 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.102938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254e09c-d621-40b4-a907-af7a67fe9872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.122372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:06.132992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4254e09c-d621-40b4-a907-af7a67fe9872 None None] MFA Rules not processed for user `38a31ab7e2fa4ca89ad081ab4871cd5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:06.134196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4254e09c-d621-40b4-a907-af7a67fe9872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:06.202237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1262/2530] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:06.213398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:06.215049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:06.272117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0470575d-7a27-4adb-8c0d-382dce1ea7d9 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:06.275596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd55089-fa25-43e0-bc43-ef171927d952 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:06.276222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd55089-fa25-43e0-bc43-ef171927d952 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.276496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd55089-fa25-43e0-bc43-ef171927d952 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.276763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd55089-fa25-43e0-bc43-ef171927d952 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.285347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd55089-fa25-43e0-bc43-ef171927d952 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:06.286757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd55089-fa25-43e0-bc43-ef171927d952 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:06.295880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1269/2531] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:06.315801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254e09c-d621-40b4-a907-af7a67fe9872 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:06.377756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254e09c-d621-40b4-a907-af7a67fe9872 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:06.382361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:06.383002 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.383283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.383554 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] PATH_INFO: `/v3/users/38a31ab7e2fa4ca89ad081ab4871cd5b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.384700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=38a31ab7e2fa4ca89ad081ab4871cd5b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:06.386666 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:06.436325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38a31ab7e2fa4ca89ad081ab4871cd5b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:06.436982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38a31ab7e2fa4ca89ad081ab4871cd5b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:06.443918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38a31ab7e2fa4ca89ad081ab4871cd5b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:06.444579 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38a31ab7e2fa4ca89ad081ab4871cd5b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:06.456682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38a31ab7e2fa4ca89ad081ab4871cd5b'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:06.469196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1263/2532] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:01:06 2022] DELETE /identity/v3/users/38a31ab7e2fa4ca89ad081ab4871cd5b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:06.473614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.473907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.474185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.493428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:06.494625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:06.555334 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1270/2533] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:06.565900 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:06.567412 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:06.587110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:06.625668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e124e466-6916-4a40-a046-1066ac4c7e2b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:06.629557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903e2177-28d3-401e-9233-875f44bb7bb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:06.630148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e2177-28d3-401e-9233-875f44bb7bb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.630428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e2177-28d3-401e-9233-875f44bb7bb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.630696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903e2177-28d3-401e-9233-875f44bb7bb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.639532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903e2177-28d3-401e-9233-875f44bb7bb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:06.641028 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903e2177-28d3-401e-9233-875f44bb7bb7 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:06.650408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1264/2534] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:06.917304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:06.919414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:06.933277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad0bfd6-56a1-4923-ab08-08a52dbdd4d6 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:06.939955 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:06.940941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:06.941471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:06.941839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] PATH_INFO: `/v3/projects/5c5e62bd9fa245908de33d5f0a954670` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:06.943765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c5e62bd9fa245908de33d5f0a954670)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:06.945754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:06.992949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c5e62bd9fa245908de33d5f0a954670', 'request_id': 'req-def565b3-1717-48e8-906f-3ea577e06d37'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:06.999584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c5e62bd9fa245908de33d5f0a954670', 'request_id': 'req-def565b3-1717-48e8-906f-3ea577e06d37'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:07.005376 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c5e62bd9fa245908de33d5f0a954670', 'request_id': 'req-def565b3-1717-48e8-906f-3ea577e06d37'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:07.013527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:07.019014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] The token cache is being invalidate because project 5c5e62bd9fa245908de33d5f0a954670 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:01:07.019440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:07.020369 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-def565b3-1717-48e8-906f-3ea577e06d37 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:07.022182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1271/2535] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:01:06 2022] DELETE /identity/v3/projects/5c5e62bd9fa245908de33d5f0a954670 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:07.026793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a637af0b-41ba-44f7-b023-63dd6180e58a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.027270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a637af0b-41ba-44f7-b023-63dd6180e58a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.027402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a637af0b-41ba-44f7-b023-63dd6180e58a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.048618 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a637af0b-41ba-44f7-b023-63dd6180e58a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:07.049800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a637af0b-41ba-44f7-b023-63dd6180e58a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:07.108008 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1265/2536] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:07.115842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167683b7-5ae5-4704-a9ae-d2d5f0d06a95 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.116163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167683b7-5ae5-4704-a9ae-d2d5f0d06a95 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.116441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167683b7-5ae5-4704-a9ae-d2d5f0d06a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.135993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-167683b7-5ae5-4704-a9ae-d2d5f0d06a95 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:07.137242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-167683b7-5ae5-4704-a9ae-d2d5f0d06a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:07.144904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1272/2537] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:07.148927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a637af0b-41ba-44f7-b023-63dd6180e58a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:07.160492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a637af0b-41ba-44f7-b023-63dd6180e58a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:07.165262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:07.165874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.166147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.166411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.167042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:01:07.167337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:07.170021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:07.181548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1266/2538] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:01:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:01:07.186279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167683b7-5ae5-4704-a9ae-d2d5f0d06a95 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:07.196527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167683b7-5ae5-4704-a9ae-d2d5f0d06a95 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:07.201256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:07.201845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.202125 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.202397 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.203015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:07.204939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:07.233317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1273/2539] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:01:07 2022] POST /identity/v3/projects => generated 365 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:01:07.238725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:07.249766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf95ce5-1f6e-4eb7-8e39-f6e76347d96b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:07.308253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:07.308854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.309134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.309405 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.310042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:07.311965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:07.336424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1267/2540] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:01:07 2022] POST /identity/v3/users => generated 283 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:01:07.337614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:07.348373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f3902e-07ea-4feb-9022-42eaff5e9d91 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:07.352881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:07.353464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.353742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.354011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.354643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:07.356400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:07.366166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1274/2541] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:07.369924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:07.380283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc5983b-6b80-416c-894b-6377ad200251 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:07.384852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c66f795-56bd-4c8f-a1dd-34134b133581 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:07.385425 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c66f795-56bd-4c8f-a1dd-34134b133581 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.385704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c66f795-56bd-4c8f-a1dd-34134b133581 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.385976 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c66f795-56bd-4c8f-a1dd-34134b133581 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.386637 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c66f795-56bd-4c8f-a1dd-34134b133581 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:07.388941 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c66f795-56bd-4c8f-a1dd-34134b133581 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:07.399251 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1268/2542] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:07.403917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:07.413978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8490bbd1-596a-457c-b4df-539bb6b8f636 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:07.418913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:07.419598 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.419877 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.420183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] PATH_INFO: `/v3/projects/72e82780f8154e0f89657eaa6caebc1e/users/36345e6240ea4a58b360442f2ff3f5a2/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.434318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72e82780f8154e0f89657eaa6caebc1e, user_id=36345e6240ea4a58b360442f2ff3f5a2, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:07.437415 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:07.449448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1275/2543] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:01:07 2022] PUT /identity/v3/projects/72e82780f8154e0f89657eaa6caebc1e/users/36345e6240ea4a58b360442f2ff3f5a2/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:07.461968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a975ea6a-03a5-4efd-8229-25284a051796 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.462250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a975ea6a-03a5-4efd-8229-25284a051796 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.462517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a975ea6a-03a5-4efd-8229-25284a051796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.493291 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a975ea6a-03a5-4efd-8229-25284a051796 None None] MFA Rules not processed for user `36345e6240ea4a58b360442f2ff3f5a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:07.494487 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a975ea6a-03a5-4efd-8229-25284a051796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:07.511418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:07.565024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1269/2544] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:07 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:07.577348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:07.579097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:07.636112 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50898f19-8c5a-4847-bd20-ca61818b3c50 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:07.640637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:07.641244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:07.641521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:07.641791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:07.650118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:07.651553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:07.661207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1276/2545] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:07 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:17.562175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=20 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:17.657658 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=54,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:39.128747 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9914d2df-504a-4de4-bb06-42326a555699 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:39.129277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9914d2df-504a-4de4-bb06-42326a555699 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:39.129277 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9914d2df-504a-4de4-bb06-42326a555699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:39.148877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-9914d2df-504a-4de4-bb06-42326a555699 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:39.150373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-9914d2df-504a-4de4-bb06-42326a555699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:39.206233 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1270/2546] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 02:01:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:39.252866 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:39.254252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:39.329102 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003d3572-2642-493e-906f-be792a22aeb6 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:39.334123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc9d3a5-73d4-444d-b0f4-397eaa1d3f39 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:39.334705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc9d3a5-73d4-444d-b0f4-397eaa1d3f39 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:39.334982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc9d3a5-73d4-444d-b0f4-397eaa1d3f39 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:39.335251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc9d3a5-73d4-444d-b0f4-397eaa1d3f39 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:39.343134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc9d3a5-73d4-444d-b0f4-397eaa1d3f39 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:39.344728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc9d3a5-73d4-444d-b0f4-397eaa1d3f39 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:39.353317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1277/2547] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 02:01:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 02:01:49.202074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=25 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:49.350143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=32 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:53.748548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9914d2df-504a-4de4-bb06-42326a555699 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:53.859348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9914d2df-504a-4de4-bb06-42326a555699 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:53.864302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:53.864929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:53.865209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:53.865481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] PATH_INFO: `/v3/users/36345e6240ea4a58b360442f2ff3f5a2` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:53.866627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=36345e6240ea4a58b360442f2ff3f5a2)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:53.869466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:53.918533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36345e6240ea4a58b360442f2ff3f5a2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:53.919175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36345e6240ea4a58b360442f2ff3f5a2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:53.926556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36345e6240ea4a58b360442f2ff3f5a2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:53.926855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36345e6240ea4a58b360442f2ff3f5a2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:53.938674 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36345e6240ea4a58b360442f2ff3f5a2'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:53.951761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1271/2548] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:01:53 2022] DELETE /identity/v3/users/36345e6240ea4a58b360442f2ff3f5a2 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:53.956257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:53.956744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:53.956850 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.006886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:54.008038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:54.070280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1278/2549] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:54.080016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:54.081399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:54.124165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:54.160047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589dd720-0992-4e29-827e-f271526a92f1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:54.164283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bd02a5-4f5d-4267-bff2-e1938ab57b72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:54.164889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd02a5-4f5d-4267-bff2-e1938ab57b72 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.165572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd02a5-4f5d-4267-bff2-e1938ab57b72 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.165572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd02a5-4f5d-4267-bff2-e1938ab57b72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.175280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bd02a5-4f5d-4267-bff2-e1938ab57b72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:54.177280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bd02a5-4f5d-4267-bff2-e1938ab57b72 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:54.186902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1272/2550] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:54.486327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:54.487769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:54.500887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4379256-413f-4463-adf0-4fdacf608895 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:54.506093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:54.506719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.507008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.507282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] PATH_INFO: `/v3/projects/72e82780f8154e0f89657eaa6caebc1e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.508470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] RBAC: Authorizing `identity:delete_project(project_id=72e82780f8154e0f89657eaa6caebc1e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:54.510373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:54.562763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72e82780f8154e0f89657eaa6caebc1e', 'request_id': 'req-ff6d341d-a344-4f91-9554-6ec83247b4af'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:54.570222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72e82780f8154e0f89657eaa6caebc1e', 'request_id': 'req-ff6d341d-a344-4f91-9554-6ec83247b4af'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:54.575929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72e82780f8154e0f89657eaa6caebc1e', 'request_id': 'req-ff6d341d-a344-4f91-9554-6ec83247b4af'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:54.589649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] The token cache is being invalidate because project 72e82780f8154e0f89657eaa6caebc1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:01:54.589933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:54.590548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-ff6d341d-a344-4f91-9554-6ec83247b4af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:01:54.592136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1279/2551] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:01:54 2022] DELETE /identity/v3/projects/72e82780f8154e0f89657eaa6caebc1e => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:54.596993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c724e-605b-4ae9-8bd8-8e72e07e5c58 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.597460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c724e-605b-4ae9-8bd8-8e72e07e5c58 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.597869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0c724e-605b-4ae9-8bd8-8e72e07e5c58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.648627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-db0c724e-605b-4ae9-8bd8-8e72e07e5c58 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:54.650019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-db0c724e-605b-4ae9-8bd8-8e72e07e5c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:54.711469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1273/2552] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:54.719007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca9e20b-3e10-4f55-9ddd-5fe24a69be3d None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.719303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca9e20b-3e10-4f55-9ddd-5fe24a69be3d None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.719561 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca9e20b-3e10-4f55-9ddd-5fe24a69be3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.740408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8ca9e20b-3e10-4f55-9ddd-5fe24a69be3d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:54.741632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8ca9e20b-3e10-4f55-9ddd-5fe24a69be3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:54.749626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1280/2553] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:54.755121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0c724e-605b-4ae9-8bd8-8e72e07e5c58 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:54.765071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0c724e-605b-4ae9-8bd8-8e72e07e5c58 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:54.771583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:54.772211 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.772490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.772757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.773400 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:01:54.773693 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:54.775341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:54.787057 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1274/2554] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:01:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:01:54.791661 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca9e20b-3e10-4f55-9ddd-5fe24a69be3d None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:54.805727 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca9e20b-3e10-4f55-9ddd-5fe24a69be3d None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:54.811126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:54.811748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.812114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.812463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.813148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:54.815114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:54.844311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1281/2555] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:01:54 2022] POST /identity/v3/projects => generated 381 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:01:54.848955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:54.858828 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a00a517-b1bb-4bf4-a192-980d91542696 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:54.916128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:54.916725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.917004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.917274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.917901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:54.919820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:54.943023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1275/2556] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:01:54 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:01:54.947208 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:54.957708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f9a1c5-e3a9-43f7-8848-7c5f2ed3671b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:54.962938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:54.963537 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.963825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.964127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.964771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:54.966372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:54.979583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1282/2557] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:01:54.981140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:54.992455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31b6983-d096-4f0e-a5ee-cfd0d8fd31a4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:54.997169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1831fed-b43f-4d4b-b0ed-dc48bccbe766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:54.997757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1831fed-b43f-4d4b-b0ed-dc48bccbe766 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:54.998038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1831fed-b43f-4d4b-b0ed-dc48bccbe766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:54.998310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1831fed-b43f-4d4b-b0ed-dc48bccbe766 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:54.998952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1831fed-b43f-4d4b-b0ed-dc48bccbe766 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:55.000569 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1831fed-b43f-4d4b-b0ed-dc48bccbe766 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:55.012244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1276/2558] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:01:55.016533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:55.027197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea13ac-f983-471f-b399-e4396a517f15 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:55.031256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:55.031905 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.032268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.032546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] PATH_INFO: `/v3/projects/0b14dad85f52497c9819f0f3bb397f18/users/a14f57dd673f480c9984491a920fcacd/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.048126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b14dad85f52497c9819f0f3bb397f18, user_id=a14f57dd673f480c9984491a920fcacd, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:55.051386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:55.053886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:55.068107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1283/2559] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:01:55 2022] PUT /identity/v3/projects/0b14dad85f52497c9819f0f3bb397f18/users/a14f57dd673f480c9984491a920fcacd/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:01:55.073203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e06b6b-6c9c-42cb-8b92-3fb509becc00 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.073516 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e06b6b-6c9c-42cb-8b92-3fb509becc00 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.073790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e06b6b-6c9c-42cb-8b92-3fb509becc00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.087973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:55.094304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-02e06b6b-6c9c-42cb-8b92-3fb509becc00 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:55.095590 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-02e06b6b-6c9c-42cb-8b92-3fb509becc00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:55.165247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1277/2560] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:55.174716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:55.177016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:55.197225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b3c13-50f8-4eee-a73b-bb357a5ac998 None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.197605 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b3c13-50f8-4eee-a73b-bb357a5ac998 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.198384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b3c13-50f8-4eee-a73b-bb357a5ac998 None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.200446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1278/2561] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 02:01:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 16 02:01:55.207184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4444c5-7d2e-49b3-b27f-19af935f9d74 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.207996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4444c5-7d2e-49b3-b27f-19af935f9d74 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.208272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4444c5-7d2e-49b3-b27f-19af935f9d74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.244867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3612f009-9bf6-4379-960a-1c384e53e338 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:55.249657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:55.249657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.249918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.250008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.253070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-7b4444c5-7d2e-49b3-b27f-19af935f9d74 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:55.254998 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-7b4444c5-7d2e-49b3-b27f-19af935f9d74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:55.263968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:55.265677 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:55.276082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1284/2562] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:55.373885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1279/2563] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:01:55.391040 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:55.392697 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:55.467326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ce3e3-91b5-40a4-b3df-07bc20a89f31 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:55.472256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac14ff-9912-4089-a32c-5ca1778d4a63 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:55.473848 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac14ff-9912-4089-a32c-5ca1778d4a63 service glance] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.474171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac14ff-9912-4089-a32c-5ca1778d4a63 service glance] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.474957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac14ff-9912-4089-a32c-5ca1778d4a63 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.489454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cac14ff-9912-4089-a32c-5ca1778d4a63 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:55.492191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cac14ff-9912-4089-a32c-5ca1778d4a63 service glance] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:55.507045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1285/2564] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:55.596460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c578924a-4f77-4f34-818c-6749f3900f1a None None] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.596771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c578924a-4f77-4f34-818c-6749f3900f1a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.597039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c578924a-4f77-4f34-818c-6749f3900f1a None None] PATH_INFO: `/` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.598427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1280/2565] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 02:01:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 02:01:55.603346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3448b65-d687-472a-95bd-6c918d77ad51 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.603631 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3448b65-d687-472a-95bd-6c918d77ad51 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.604567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3448b65-d687-472a-95bd-6c918d77ad51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.625350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b3448b65-d687-472a-95bd-6c918d77ad51 None None] MFA Rules not processed for user `6a343a260fa2415e84caf959933a6c13`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:55.626832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b3448b65-d687-472a-95bd-6c918d77ad51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:55.634206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1286/2566] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:55.647410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c578924a-4f77-4f34-818c-6749f3900f1a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:55.652151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c578924a-4f77-4f34-818c-6749f3900f1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:55.668661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c578924a-4f77-4f34-818c-6749f3900f1a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:55.672474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7218ea1d-9ff2-4d51-b825-b94720e9a3d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:55.673148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7218ea1d-9ff2-4d51-b825-b94720e9a3d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.673503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7218ea1d-9ff2-4d51-b825-b94720e9a3d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.673856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7218ea1d-9ff2-4d51-b825-b94720e9a3d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.682102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7218ea1d-9ff2-4d51-b825-b94720e9a3d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:55.683623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7218ea1d-9ff2-4d51-b825-b94720e9a3d1 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:55.692920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1281/2567] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:55.780902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f64f8f8-bde3-4737-b412-2318eeb2c619 None None] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.781187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f64f8f8-bde3-4737-b412-2318eeb2c619 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.781453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f64f8f8-bde3-4737-b412-2318eeb2c619 None None] PATH_INFO: `/` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.783058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1287/2568] 10.222.0.42 () {54 vars in 921 bytes} [Fri Sep 16 02:01:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 16 02:01:55.789163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cefa330-8849-47e2-8e9a-2ddc3a519c01 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.789648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cefa330-8849-47e2-8e9a-2ddc3a519c01 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.789831 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cefa330-8849-47e2-8e9a-2ddc3a519c01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.815556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4cefa330-8849-47e2-8e9a-2ddc3a519c01 None None] MFA Rules not processed for user `603c81d781754201a5ac5b7d9b84fab6`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:55.817519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4cefa330-8849-47e2-8e9a-2ddc3a519c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:55.828700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1282/2569] 10.222.0.42 () {58 vars in 1018 bytes} [Fri Sep 16 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:55.840223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f64f8f8-bde3-4737-b412-2318eeb2c619 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:55.852166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f64f8f8-bde3-4737-b412-2318eeb2c619 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:55.855685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:55.856333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.856614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.856882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:55.868251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:55.869687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:55.882371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1288/2570] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:01:55.976871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f43c36-2253-4121-a83b-b40cf8c8648c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:55.977267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f43c36-2253-4121-a83b-b40cf8c8648c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:55.977810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f43c36-2253-4121-a83b-b40cf8c8648c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:56.018823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-81f43c36-2253-4121-a83b-b40cf8c8648c None None] MFA Rules not processed for user `a14f57dd673f480c9984491a920fcacd`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:56.020753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-81f43c36-2253-4121-a83b-b40cf8c8648c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:56.060321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:56.104199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1283/2571] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:55 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:56.117484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:56.118977 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:56.200820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3cb3ca-bf98-4465-b2e4-f8dd44b7c413 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:56.204896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d11711-a1ca-499c-8c1e-4bd5be53f0d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:56.205783 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d11711-a1ca-499c-8c1e-4bd5be53f0d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:56.206227 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d11711-a1ca-499c-8c1e-4bd5be53f0d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:56.206596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d11711-a1ca-499c-8c1e-4bd5be53f0d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:56.218947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d11711-a1ca-499c-8c1e-4bd5be53f0d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:56.220534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d11711-a1ca-499c-8c1e-4bd5be53f0d9 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:56.241901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1289/2572] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:56 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 02:01:57.652370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559b4d58-7be3-44aa-a452-5b0b8a4d418f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:57.652862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559b4d58-7be3-44aa-a452-5b0b8a4d418f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:57.653063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559b4d58-7be3-44aa-a452-5b0b8a4d418f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:57.677127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-559b4d58-7be3-44aa-a452-5b0b8a4d418f None None] MFA Rules not processed for user `a14f57dd673f480c9984491a920fcacd`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:57.678718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-559b4d58-7be3-44aa-a452-5b0b8a4d418f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:57.686801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1284/2573] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:01:57 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:57.846027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:57.846511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:57.846702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:57.866925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:01:57.868165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:57.888352 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:01:57.929615 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1290/2574] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 02:01:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:01:57.934633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559b4d58-7be3-44aa-a452-5b0b8a4d418f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:57.942496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:01:57.943923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:57.945382 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559b4d58-7be3-44aa-a452-5b0b8a4d418f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:57.949018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:57.949609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:57.949889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:57.950153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:57.951910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:01:57.961986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:57.963415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:57.971345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1285/2575] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:01:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 02:01:58.015663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fc8214-6570-4899-ae8a-3a6016fdc728 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:01:58.021463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2dde111-c6db-4779-abdc-d4e02652c222 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:01:58.022070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dde111-c6db-4779-abdc-d4e02652c222 service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:01:58.022349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dde111-c6db-4779-abdc-d4e02652c222 service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:01:58.022619 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2dde111-c6db-4779-abdc-d4e02652c222 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:01:58.030846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2dde111-c6db-4779-abdc-d4e02652c222 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:01:58.032482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2dde111-c6db-4779-abdc-d4e02652c222 service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:01:58.041540 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1291/2576] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 02:01:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 02:02:07.967776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=21 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:02:08.038207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=47 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:02:11.940436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:02:11.942226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:02:12.041609 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6ba0f4-e0cd-4e5b-a55a-537316bc05a7 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:02:12.047073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:02:12.047673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:02:12.047974 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:02:12.048248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:02:12.049966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:02:12.062725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:02:12.064349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:02:12.086581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1286/2577] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:02:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:02:19.779649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee0b09e-6826-4f8c-ac53-d32c7ad7ab74 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:02:19.780167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee0b09e-6826-4f8c-ac53-d32c7ad7ab74 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:02:19.780409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee0b09e-6826-4f8c-ac53-d32c7ad7ab74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:02:19.806566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-eee0b09e-6826-4f8c-ac53-d32c7ad7ab74 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:02:19.807772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-eee0b09e-6826-4f8c-ac53-d32c7ad7ab74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:02:19.815641 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1292/2578] 10.222.0.42 () {58 vars in 967 bytes} [Fri Sep 16 02:02:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:02:19.827727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:02:19.839314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468906c7-9a38-4bcb-b213-c20e17b7a89d service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:02:19.844326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:02:19.844920 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:02:19.845192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:02:19.845456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:02:19.854628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:02:19.856228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:02:19.864284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1287/2579] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 02:02:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 16 02:02:29.795667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:02:29.860971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=47 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:03.451623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee0b09e-6826-4f8c-ac53-d32c7ad7ab74 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:03.470989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee0b09e-6826-4f8c-ac53-d32c7ad7ab74 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:03.474627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:03.475225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:03.475507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:03.475779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:03.477527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:03.490996 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:03.492468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:03.502733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1293/2580] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:06.497250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:06.498787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:06.514062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e886337-5786-49e8-94d5-ac7e65b50b5d service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:06.527911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:06.528981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:06.529330 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:06.529652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:06.531616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:06.545140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:06.546805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:06.559853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1288/2581] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:06 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:06.569199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:06.571005 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:06.588108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fc0ba5-923f-43a0-8178-79702aaad8e9 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:06.594748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:06.595607 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:06.595983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:06.596269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] PATH_INFO: `/v3/users/6fc9485505c443acb946ec66e0018f7b` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:06.597413 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] RBAC: Authorizing `identity:delete_user(user_id=6fc9485505c443acb946ec66e0018f7b)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:06.599510 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:06.654691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6fc9485505c443acb946ec66e0018f7b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:06.655340 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6fc9485505c443acb946ec66e0018f7b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:06.656025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6fc9485505c443acb946ec66e0018f7b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:06.663181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6fc9485505c443acb946ec66e0018f7b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:06.676165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-854d94a7-dd28-477e-adce-99f50af0740f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6fc9485505c443acb946ec66e0018f7b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:06.688863 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1294/2582] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:06 2022] DELETE /identity/v3/users/6fc9485505c443acb946ec66e0018f7b => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:06.866878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:06.882892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8539f8be-8247-48ac-93ae-414b37a8e806 service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:06.941473 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:06.942089 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:06.942363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:06.942627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] PATH_INFO: `/v3/projects/eddbb633debb4badb74f142d92baed98` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:06.943708 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eddbb633debb4badb74f142d92baed98)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:06.945624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:06.991261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eddbb633debb4badb74f142d92baed98', 'request_id': 'req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:06.998142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eddbb633debb4badb74f142d92baed98', 'request_id': 'req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:07.003774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eddbb633debb4badb74f142d92baed98', 'request_id': 'req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:07.016000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] The token cache is being invalidate because project eddbb633debb4badb74f142d92baed98 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:07.016303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:07.016926 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6c2d9ea9-6a82-4dbd-8f8f-03ad0426bcc9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:07.018388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1289/2583] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:06 2022] DELETE /identity/v3/projects/eddbb633debb4badb74f142d92baed98 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:07.023240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab8f5e-c031-426e-a9e1-2827a4f2f36b None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.023525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab8f5e-c031-426e-a9e1-2827a4f2f36b None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.023793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dab8f5e-c031-426e-a9e1-2827a4f2f36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.048796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6dab8f5e-c031-426e-a9e1-2827a4f2f36b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:07.050255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6dab8f5e-c031-426e-a9e1-2827a4f2f36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:07.109090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1295/2584] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:07.117254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494fb28-876c-4b38-b7ee-4034f1b63a6b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.117538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494fb28-876c-4b38-b7ee-4034f1b63a6b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.117811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494fb28-876c-4b38-b7ee-4034f1b63a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.137965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e494fb28-876c-4b38-b7ee-4034f1b63a6b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:07.139071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e494fb28-876c-4b38-b7ee-4034f1b63a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:07.146911 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1290/2585] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:07.151145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab8f5e-c031-426e-a9e1-2827a4f2f36b None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.160733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dab8f5e-c031-426e-a9e1-2827a4f2f36b None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.165213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.165803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.166080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.166347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.166994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:03:07.168039 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.169613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.180867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1296/2586] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:03:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 02:03:07.185629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e494fb28-876c-4b38-b7ee-4034f1b63a6b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.194574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e494fb28-876c-4b38-b7ee-4034f1b63a6b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.199379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.200009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.200284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.200547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.201154 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.203026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.231080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1291/2587] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/projects => generated 405 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:07.235839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.246333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff311042-c8b3-43a4-a579-9d66ec822a9f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.282037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:07.307463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.308193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.308474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.308742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.309358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.311255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.332162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1297/2588] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/users => generated 303 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:03:07.336324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.347016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fde756-bfe0-456e-b6f5-c215b902b10d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.351630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.352253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.352534 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.352804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.353440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.355768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.365526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1292/2589] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:07.369316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.379319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0d4b7a-a24e-48a8-abba-b9dfbb50c4d7 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.383875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a450e7e-7789-4fcf-8484-c932d0713f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.384481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a450e7e-7789-4fcf-8484-c932d0713f31 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.384759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a450e7e-7789-4fcf-8484-c932d0713f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.385057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a450e7e-7789-4fcf-8484-c932d0713f31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.385687 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a450e7e-7789-4fcf-8484-c932d0713f31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.387493 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a450e7e-7789-4fcf-8484-c932d0713f31 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.397313 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1298/2590] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:07.401343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.411761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04448d86-ad89-478c-8e59-8bd98c2259c9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.416376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.417004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.417919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.418196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] PATH_INFO: `/v3/projects/eb1f5061322a475fbd430f4d3004bef3/users/004b3a6ebae247b9a6c1482f392c71e6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.432545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb1f5061322a475fbd430f4d3004bef3, user_id=004b3a6ebae247b9a6c1482f392c71e6, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.435354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.448383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1293/2591] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:07 2022] PUT /identity/v3/projects/eb1f5061322a475fbd430f4d3004bef3/users/004b3a6ebae247b9a6c1482f392c71e6/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:03:07.459845 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c655dd0-fd64-4808-8907-6a8165fc2985 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.460166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c655dd0-fd64-4808-8907-6a8165fc2985 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.460439 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c655dd0-fd64-4808-8907-6a8165fc2985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.488839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-3c655dd0-fd64-4808-8907-6a8165fc2985 None None] MFA Rules not processed for user `004b3a6ebae247b9a6c1482f392c71e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:07.490026 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3c655dd0-fd64-4808-8907-6a8165fc2985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:07.554885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1299/2592] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:07.559718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.572846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:07.622633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cca8ab0-dddd-47c3-980e-f5c1023c48ba admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.627523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.628146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.628428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.628703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.629321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.631243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.658571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1294/2593] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/projects => generated 405 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:07.662791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c655dd0-fd64-4808-8907-6a8165fc2985 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.673629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c655dd0-fd64-4808-8907-6a8165fc2985 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.729098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.729733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.730014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.730283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.730903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.732864 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.753651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1300/2594] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/users => generated 309 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:07.757787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.768542 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f41dea4-d8e9-4963-9572-2bdefe24dfc8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.773111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.773701 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.773981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.774248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.774879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.777146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.787892 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1295/2595] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:07.792599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.803308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e90ce11-3123-4fac-8354-a5becf16a7e4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.808246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.809270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.809270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.809483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] PATH_INFO: `/v3/projects/0bfe70e6a9354196b7e0de4e917ac3da/users/8895fefbee40461ab29b77a1f7ca5dd3/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.825742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0bfe70e6a9354196b7e0de4e917ac3da, user_id=8895fefbee40461ab29b77a1f7ca5dd3, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.828402 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.842146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1301/2596] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:07 2022] PUT /identity/v3/projects/0bfe70e6a9354196b7e0de4e917ac3da/users/8895fefbee40461ab29b77a1f7ca5dd3/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:07.846428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.860855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727a0c7-f808-4f71-8319-1bbf11940b1a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.919929 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83f6886-a40d-4a08-87fb-09a3932f329a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.920139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f6886-a40d-4a08-87fb-09a3932f329a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.920356 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f6886-a40d-4a08-87fb-09a3932f329a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.920624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83f6886-a40d-4a08-87fb-09a3932f329a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.921269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83f6886-a40d-4a08-87fb-09a3932f329a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.922852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83f6886-a40d-4a08-87fb-09a3932f329a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.936859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1296/2597] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:07.940323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:07.952566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e6763b-b38d-4060-bc63-f1fd3492f3c6 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:07.957266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:07.957913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.958187 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.958457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] PATH_INFO: `/v3/projects/0bfe70e6a9354196b7e0de4e917ac3da/users/8895fefbee40461ab29b77a1f7ca5dd3/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:07.960380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0bfe70e6a9354196b7e0de4e917ac3da, user_id=8895fefbee40461ab29b77a1f7ca5dd3, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:07.962994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:07.970028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:07.979062 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1302/2598] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:07 2022] PUT /identity/v3/projects/0bfe70e6a9354196b7e0de4e917ac3da/users/8895fefbee40461ab29b77a1f7ca5dd3/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:03:07.989755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d6b73-e3dd-4aeb-acc9-d6bac66bcb7e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:07.990043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d6b73-e3dd-4aeb-acc9-d6bac66bcb7e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:07.990370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96d6b73-e3dd-4aeb-acc9-d6bac66bcb7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:08.019668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-d96d6b73-e3dd-4aeb-acc9-d6bac66bcb7e None None] MFA Rules not processed for user `8895fefbee40461ab29b77a1f7ca5dd3`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:08.020918 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d96d6b73-e3dd-4aeb-acc9-d6bac66bcb7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:08.088200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1297/2599] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:07 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:08.099536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:08.101576 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:08.180675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5698ba-7570-472b-94c2-3fa209968c01 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:08.184234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e45767-583c-4993-8535-5f996f6b8034 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:08.184831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e45767-583c-4993-8535-5f996f6b8034 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:08.185108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e45767-583c-4993-8535-5f996f6b8034 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:08.185378 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e45767-583c-4993-8535-5f996f6b8034 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:08.194360 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e45767-583c-4993-8535-5f996f6b8034 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:08.195778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e45767-583c-4993-8535-5f996f6b8034 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:08.205222 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1303/2600] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:08 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:08.362313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96d6b73-e3dd-4aeb-acc9-d6bac66bcb7e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:08.372496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:08.431785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96d6b73-e3dd-4aeb-acc9-d6bac66bcb7e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:08.437586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:08.438301 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:08.438575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:08.438843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] PATH_INFO: `/v3/users/004b3a6ebae247b9a6c1482f392c71e6` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:08.440004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=004b3a6ebae247b9a6c1482f392c71e6)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:08.441969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:08.491587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '004b3a6ebae247b9a6c1482f392c71e6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:08.492430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '004b3a6ebae247b9a6c1482f392c71e6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:08.500042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '004b3a6ebae247b9a6c1482f392c71e6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:08.500689 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '004b3a6ebae247b9a6c1482f392c71e6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:08.513326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '004b3a6ebae247b9a6c1482f392c71e6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:08.526332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1298/2601] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:08 2022] DELETE /identity/v3/users/004b3a6ebae247b9a6c1482f392c71e6 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:08.531724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:08.532044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:08.532338 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:08.551214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:08.552429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:08.611346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1304/2602] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:08.620977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:08.622353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:08.695676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e8b02f-94c0-4030-b29f-c2990d8c88bc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:08.699891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:08.700507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:08.700782 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:08.701050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:08.709868 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:08.711284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:08.721406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1299/2603] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:09.007781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:09.009314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:09.039786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0eca9-fd46-46f5-87d6-86d082f31324 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:09.046238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:09.047151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.047544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.047743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] PATH_INFO: `/v3/projects/eb1f5061322a475fbd430f4d3004bef3` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.048963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb1f5061322a475fbd430f4d3004bef3)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:09.051000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:09.099008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb1f5061322a475fbd430f4d3004bef3', 'request_id': 'req-047004af-8b51-4cac-bb23-28975a4bd3cf'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.105346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb1f5061322a475fbd430f4d3004bef3', 'request_id': 'req-047004af-8b51-4cac-bb23-28975a4bd3cf'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.110860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb1f5061322a475fbd430f4d3004bef3', 'request_id': 'req-047004af-8b51-4cac-bb23-28975a4bd3cf'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.123349 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] The token cache is being invalidate because project eb1f5061322a475fbd430f4d3004bef3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:09.123638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.124274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.126110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1305/2604] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:09 2022] DELETE /identity/v3/projects/eb1f5061322a475fbd430f4d3004bef3 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:09.130342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:09.140413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8df7bd8-9b9b-4c55-a6c0-c6cb44116220 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:09.157220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:09.199220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:09.200056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.200335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.200603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] PATH_INFO: `/v3/users/8895fefbee40461ab29b77a1f7ca5dd3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.201715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8895fefbee40461ab29b77a1f7ca5dd3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:09.203636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:09.251427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8895fefbee40461ab29b77a1f7ca5dd3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.252007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8895fefbee40461ab29b77a1f7ca5dd3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.258676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8895fefbee40461ab29b77a1f7ca5dd3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.259288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8895fefbee40461ab29b77a1f7ca5dd3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.271256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cb950e29-af89-442c-bfc4-887151220645 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8895fefbee40461ab29b77a1f7ca5dd3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.286723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1300/2605] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:09 2022] DELETE /identity/v3/users/8895fefbee40461ab29b77a1f7ca5dd3 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:09.551897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:09.553448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:09.563761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:09.568077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047004af-8b51-4cac-bb23-28975a4bd3cf admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:09.626581 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:09.627203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.627482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.627757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] PATH_INFO: `/v3/projects/0bfe70e6a9354196b7e0de4e917ac3da` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.628884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0bfe70e6a9354196b7e0de4e917ac3da)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:09.630762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:09.675249 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0bfe70e6a9354196b7e0de4e917ac3da', 'request_id': 'req-027543ae-3a78-4144-ad14-e2e8980e3483'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.681406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0bfe70e6a9354196b7e0de4e917ac3da', 'request_id': 'req-027543ae-3a78-4144-ad14-e2e8980e3483'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.687445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0bfe70e6a9354196b7e0de4e917ac3da', 'request_id': 'req-027543ae-3a78-4144-ad14-e2e8980e3483'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.700233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] The token cache is being invalidate because project 0bfe70e6a9354196b7e0de4e917ac3da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:09.700522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.701149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-027543ae-3a78-4144-ad14-e2e8980e3483 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:09.702633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1306/2606] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:09 2022] DELETE /identity/v3/projects/0bfe70e6a9354196b7e0de4e917ac3da => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:09.707445 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05db951-bfbd-4f50-a2ed-9f624ac79a75 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.707912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05db951-bfbd-4f50-a2ed-9f624ac79a75 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.708050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05db951-bfbd-4f50-a2ed-9f624ac79a75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.727424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f05db951-bfbd-4f50-a2ed-9f624ac79a75 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:09.728647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f05db951-bfbd-4f50-a2ed-9f624ac79a75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:09.787109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1301/2607] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:09.796507 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf71c5-12c1-46bc-b304-576926a48fc3 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.796729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf71c5-12c1-46bc-b304-576926a48fc3 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.797123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf71c5-12c1-46bc-b304-576926a48fc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.817553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-0abf71c5-12c1-46bc-b304-576926a48fc3 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:09.818752 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0abf71c5-12c1-46bc-b304-576926a48fc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:09.828449 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1307/2608] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:09.832649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05db951-bfbd-4f50-a2ed-9f624ac79a75 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:09.843184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05db951-bfbd-4f50-a2ed-9f624ac79a75 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:09.847748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:09.848376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.848656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.848925 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.849564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:03:09.849888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:09.851502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:09.863588 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1302/2609] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:03:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:03:09.868012 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf71c5-12c1-46bc-b304-576926a48fc3 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:09.878127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf71c5-12c1-46bc-b304-576926a48fc3 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:09.882725 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:09.883323 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.883597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.883858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.884546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:09.886578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:09.914801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1308/2610] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:09 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:09.919486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:09.929833 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e8eae7-49ea-4d16-9f05-6ea76e21231f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:09.954317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:09.988162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:09.988758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:09.989038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:09.989310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:09.989935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:09.991888 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.014063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1303/2611] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:09 2022] POST /identity/v3/users => generated 283 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:10.018199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.028481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad6e61-ce40-4931-ad6e-8d0238c5f472 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.033072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.033659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.033934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.034200 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.034831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.036466 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.047764 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1309/2612] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:10.051499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.062322 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8da3b7-50e8-45f9-8cc7-d7fff4f0f1f8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.066933 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962487c3-dacf-41de-8599-02e7782fe75f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.067524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962487c3-dacf-41de-8599-02e7782fe75f admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.067800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962487c3-dacf-41de-8599-02e7782fe75f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.068114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962487c3-dacf-41de-8599-02e7782fe75f admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.068752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962487c3-dacf-41de-8599-02e7782fe75f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.070333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962487c3-dacf-41de-8599-02e7782fe75f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.080993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1304/2613] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:10.085044 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.094847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3301e31-519b-408b-bc93-005370935cfa admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.099488 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.100149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.100432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.100705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] PATH_INFO: `/v3/projects/4cf4966f32e64d38a737e9626cb06aa2/users/c2bfeb27f4644985bcff996850f12931/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.115377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cf4966f32e64d38a737e9626cb06aa2, user_id=c2bfeb27f4644985bcff996850f12931, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.118034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.130952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1310/2614] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:10 2022] PUT /identity/v3/projects/4cf4966f32e64d38a737e9626cb06aa2/users/c2bfeb27f4644985bcff996850f12931/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:03:10.140904 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eea76f-69c6-4a99-9d7f-b4b4e1255c11 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.141189 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eea76f-69c6-4a99-9d7f-b4b4e1255c11 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.141459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77eea76f-69c6-4a99-9d7f-b4b4e1255c11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.171402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-77eea76f-69c6-4a99-9d7f-b4b4e1255c11 None None] MFA Rules not processed for user `c2bfeb27f4644985bcff996850f12931`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:10.172625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-77eea76f-69c6-4a99-9d7f-b4b4e1255c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:10.236558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1305/2615] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:10 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:10.240958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.273567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:10.304511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf472ebe-7df3-498b-9cc0-8dcf5e26c539 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.309485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.310081 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.310358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.310628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.311253 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.313154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.338754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1311/2616] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:10 2022] POST /identity/v3/projects => generated 363 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:10.343407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eea76f-69c6-4a99-9d7f-b4b4e1255c11 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.354377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77eea76f-69c6-4a99-9d7f-b4b4e1255c11 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.409949 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.410544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.410819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.411089 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.411704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.413636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.434486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1306/2617] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:10 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:10.438522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.448720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc75ad73-55af-4d26-ab56-20c08aa0e173 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.453318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.453900 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.454175 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.454441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.455069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.456680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.467708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1312/2618] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:10.471580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.482342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7020a45c-7ac1-4e2c-ac3d-e333dfc1450a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.487134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.487762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.488064 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.488331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] PATH_INFO: `/v3/projects/da4c6a4172ae4412a0b7871c517b2f79/users/a25b159688fc497dbeea4acf42ec7391/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.504017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da4c6a4172ae4412a0b7871c517b2f79, user_id=a25b159688fc497dbeea4acf42ec7391, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.506811 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.515259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:10.520354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1307/2619] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:10 2022] PUT /identity/v3/projects/da4c6a4172ae4412a0b7871c517b2f79/users/a25b159688fc497dbeea4acf42ec7391/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:03:10.524527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.535455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad07e85-5e71-4a5b-8d8e-20fe94d04931 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.591311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ba7bca-aaaf-4b4f-8da5-6a195d5d4d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.591844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ba7bca-aaaf-4b4f-8da5-6a195d5d4d49 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.592153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ba7bca-aaaf-4b4f-8da5-6a195d5d4d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.592425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ba7bca-aaaf-4b4f-8da5-6a195d5d4d49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.593066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ba7bca-aaaf-4b4f-8da5-6a195d5d4d49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.594651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ba7bca-aaaf-4b4f-8da5-6a195d5d4d49 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.604325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1313/2620] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:10.609256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.619217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62989e9-932a-45f9-8f1a-03cb63428a77 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.624527 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.625159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.625434 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.626086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] PATH_INFO: `/v3/projects/da4c6a4172ae4412a0b7871c517b2f79/users/a25b159688fc497dbeea4acf42ec7391/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.628053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] RBAC: Authorizing `identity:create_grant(project_id=da4c6a4172ae4412a0b7871c517b2f79, user_id=a25b159688fc497dbeea4acf42ec7391, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.630656 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.640535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1308/2621] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:10 2022] PUT /identity/v3/projects/da4c6a4172ae4412a0b7871c517b2f79/users/a25b159688fc497dbeea4acf42ec7391/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:03:10.650400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22f0ab4-08f3-4f1e-a74e-653886364c16 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.650685 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22f0ab4-08f3-4f1e-a74e-653886364c16 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.650978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22f0ab4-08f3-4f1e-a74e-653886364c16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.679910 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-d22f0ab4-08f3-4f1e-a74e-653886364c16 None None] MFA Rules not processed for user `a25b159688fc497dbeea4acf42ec7391`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:10.681182 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d22f0ab4-08f3-4f1e-a74e-653886364c16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:10.751475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1314/2622] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:10 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:10.763067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:10.764841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:10.822766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101dd5bf-ad7d-4168-9911-cf8faed522be admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:10.826413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4253948c-0455-428d-a8fe-fb5391cb1064 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:10.827009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253948c-0455-428d-a8fe-fb5391cb1064 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:10.827313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253948c-0455-428d-a8fe-fb5391cb1064 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:10.827580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4253948c-0455-428d-a8fe-fb5391cb1064 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:10.836419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4253948c-0455-428d-a8fe-fb5391cb1064 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:10.837846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4253948c-0455-428d-a8fe-fb5391cb1064 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:10.847302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1309/2623] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:10 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:11.266708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22f0ab4-08f3-4f1e-a74e-653886364c16 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:11.298733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:11.332443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22f0ab4-08f3-4f1e-a74e-653886364c16 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:11.336936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:11.337549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:11.337827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:11.338092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] PATH_INFO: `/v3/users/c2bfeb27f4644985bcff996850f12931` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:11.340237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2bfeb27f4644985bcff996850f12931)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:11.342130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:11.386699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2bfeb27f4644985bcff996850f12931'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.387426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2bfeb27f4644985bcff996850f12931'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.388056 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2bfeb27f4644985bcff996850f12931'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.395186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2bfeb27f4644985bcff996850f12931'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.407291 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2bfeb27f4644985bcff996850f12931'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.420330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1315/2624] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:11 2022] DELETE /identity/v3/users/c2bfeb27f4644985bcff996850f12931 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:11.424996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:11.425281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:11.425553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:11.445114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:11.446325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:11.506079 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1310/2625] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:11.515907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:11.517328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:11.575944 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063a6615-8311-49d4-b5b6-7743da375089 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:11.579741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:11.580366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:11.580644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:11.580911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:11.589818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:11.591233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:11.600591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1316/2626] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:11.868805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:11.870360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:11.882843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adef2bbd-da22-42c1-9519-392154451ac6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:11.887668 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:11.888329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:11.888704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:11.888874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] PATH_INFO: `/v3/projects/4cf4966f32e64d38a737e9626cb06aa2` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:11.889955 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4cf4966f32e64d38a737e9626cb06aa2)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:11.891845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:11.944406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4cf4966f32e64d38a737e9626cb06aa2', 'request_id': 'req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.950820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4cf4966f32e64d38a737e9626cb06aa2', 'request_id': 'req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.956506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4cf4966f32e64d38a737e9626cb06aa2', 'request_id': 'req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.968777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] The token cache is being invalidate because project 4cf4966f32e64d38a737e9626cb06aa2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:11.969062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.969706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:11.971178 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1311/2627] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:11 2022] DELETE /identity/v3/projects/4cf4966f32e64d38a737e9626cb06aa2 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:11.975167 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:11.985584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c28353-ef76-4971-841d-1c0033e52ef3 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.029671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:12.043573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.044220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.044500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.044771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] PATH_INFO: `/v3/users/a25b159688fc497dbeea4acf42ec7391` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.045883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a25b159688fc497dbeea4acf42ec7391)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.047784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.093457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a25b159688fc497dbeea4acf42ec7391'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.094084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a25b159688fc497dbeea4acf42ec7391'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.094676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a25b159688fc497dbeea4acf42ec7391'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.101748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a25b159688fc497dbeea4acf42ec7391'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.113890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f1fa2c10-8113-43ae-9f95-c618b338a111 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a25b159688fc497dbeea4acf42ec7391'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.126809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1317/2628] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:11 2022] DELETE /identity/v3/users/a25b159688fc497dbeea4acf42ec7391 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:12.402025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:12.403568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:12.418594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc943cd8-edab-477c-ac14-f8ab2f3a19d9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.428635 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:12.477766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.478305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.478582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.478850 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] PATH_INFO: `/v3/projects/da4c6a4172ae4412a0b7871c517b2f79` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.480159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da4c6a4172ae4412a0b7871c517b2f79)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.482059 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.529451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da4c6a4172ae4412a0b7871c517b2f79', 'request_id': 'req-29379c13-451a-4564-8a39-2fc079f3ddf6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.535785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da4c6a4172ae4412a0b7871c517b2f79', 'request_id': 'req-29379c13-451a-4564-8a39-2fc079f3ddf6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.541423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da4c6a4172ae4412a0b7871c517b2f79', 'request_id': 'req-29379c13-451a-4564-8a39-2fc079f3ddf6'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.553267 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] The token cache is being invalidate because project da4c6a4172ae4412a0b7871c517b2f79 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:12.553548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.554183 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-29379c13-451a-4564-8a39-2fc079f3ddf6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:12.555641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1312/2629] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:12 2022] DELETE /identity/v3/projects/da4c6a4172ae4412a0b7871c517b2f79 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:12.561071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13a16d-ba74-42b0-98ae-093d3e1ce021 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.561542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13a16d-ba74-42b0-98ae-093d3e1ce021 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.561639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13a16d-ba74-42b0-98ae-093d3e1ce021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.581673 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-fc13a16d-ba74-42b0-98ae-093d3e1ce021 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:12.582860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fc13a16d-ba74-42b0-98ae-093d3e1ce021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:12.639991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1318/2630] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:12.649051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc4e87-f21a-4298-8911-e6c3052e5a92 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.649377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc4e87-f21a-4298-8911-e6c3052e5a92 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.649648 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc4e87-f21a-4298-8911-e6c3052e5a92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.670644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-93fc4e87-f21a-4298-8911-e6c3052e5a92 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:12.671840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-93fc4e87-f21a-4298-8911-e6c3052e5a92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:12.680098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1313/2631] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:12.684731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc13a16d-ba74-42b0-98ae-093d3e1ce021 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:12.694302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc13a16d-ba74-42b0-98ae-093d3e1ce021 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.698904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.699484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.699759 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.700068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.700705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:03:12.701001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.702594 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.714365 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1319/2632] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:03:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:03:12.718469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fc4e87-f21a-4298-8911-e6c3052e5a92 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:12.727482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fc4e87-f21a-4298-8911-e6c3052e5a92 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.732780 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.733302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.733576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.733840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.734450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.736441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.764101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1314/2633] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:12 2022] POST /identity/v3/projects => generated 381 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:12.768161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:12.778603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032af6ff-f350-4b4a-b339-b786c34b8596 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.829927 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:12.838399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.840327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.840928 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.841372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.842177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.844316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.865239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1320/2634] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:12 2022] POST /identity/v3/users => generated 291 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:12.869316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:12.879899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d0e396-4505-47ee-8b6f-e997df0f2124 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.884526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.885111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.885390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.886225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.886858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.888482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.898426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1315/2635] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:12.903022 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:12.913183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a37ed49-d172-4cd1-9a32-91a3126f67b4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.917806 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa0722a-0256-480c-99e9-35b16bffacc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.918396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa0722a-0256-480c-99e9-35b16bffacc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.918692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa0722a-0256-480c-99e9-35b16bffacc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.918948 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa0722a-0256-480c-99e9-35b16bffacc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.919585 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa0722a-0256-480c-99e9-35b16bffacc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.921190 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa0722a-0256-480c-99e9-35b16bffacc8 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.931729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1321/2636] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:03:12.936050 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:12.946610 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a601e27-ba0c-45b7-82d3-cddf5a1afe21 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:12.951779 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:12.952458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.952738 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.953009 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] PATH_INFO: `/v3/projects/2e81b4600f474d0e9d8bc2f842ebc39e/users/e8b50f6f3bad43ad9262aa34c2399321/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:12.967830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e81b4600f474d0e9d8bc2f842ebc39e, user_id=e8b50f6f3bad43ad9262aa34c2399321, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:12.970510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:12.983055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1316/2637] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:12 2022] PUT /identity/v3/projects/2e81b4600f474d0e9d8bc2f842ebc39e/users/e8b50f6f3bad43ad9262aa34c2399321/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:12.993301 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648e376-b312-4fe1-b6a8-6f70415b9c9e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:12.993590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648e376-b312-4fe1-b6a8-6f70415b9c9e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:12.993888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9648e376-b312-4fe1-b6a8-6f70415b9c9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.021660 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-9648e376-b312-4fe1-b6a8-6f70415b9c9e None None] MFA Rules not processed for user `e8b50f6f3bad43ad9262aa34c2399321`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:13.022868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-9648e376-b312-4fe1-b6a8-6f70415b9c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:13.086305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1322/2638] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:12 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:13.091526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:13.132776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:13.155515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f92d2b-5678-41de-8006-7c4e261dd02c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:13.160212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:13.160807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.161086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.161360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.162000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:13.163885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:13.189331 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1317/2639] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:13 2022] POST /identity/v3/projects => generated 379 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:13.193563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9648e376-b312-4fe1-b6a8-6f70415b9c9e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:13.205031 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9648e376-b312-4fe1-b6a8-6f70415b9c9e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:13.262399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:13.262922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.263203 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.263474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.264137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:13.266059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:13.286771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1323/2640] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:13 2022] POST /identity/v3/users => generated 296 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:03:13.290787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:13.301148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe31dca7-5cfc-4e4d-b5b9-6a9d6e4fca96 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:13.305765 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:13.306344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.306644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.306886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.308099 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:13.309687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:13.319951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1318/2641] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:13.324495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:13.334417 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622c7345-6746-47be-9c01-5293bf839d51 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:13.338993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:13.339632 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.339912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.340225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] PATH_INFO: `/v3/projects/572f6e477d1e4c93997a2405f40e526e/users/1dd3afc910a34fa5b0a8822d748200f0/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.355915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=572f6e477d1e4c93997a2405f40e526e, user_id=1dd3afc910a34fa5b0a8822d748200f0, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:13.358582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:13.372787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1324/2642] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:13 2022] PUT /identity/v3/projects/572f6e477d1e4c93997a2405f40e526e/users/1dd3afc910a34fa5b0a8822d748200f0/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:13.376981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:13.387006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f3ba26-96c5-4d07-8509-8616e06591ef admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:13.442715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc7e4da-c380-4a9d-a28b-0d88e8b25ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:13.443337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7e4da-c380-4a9d-a28b-0d88e8b25ba1 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.443612 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7e4da-c380-4a9d-a28b-0d88e8b25ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.443879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc7e4da-c380-4a9d-a28b-0d88e8b25ba1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.444535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc7e4da-c380-4a9d-a28b-0d88e8b25ba1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:13.446108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc7e4da-c380-4a9d-a28b-0d88e8b25ba1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:13.456492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1319/2643] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:13.460662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:13.471131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1cf055-f7ac-492d-86d2-4eac27f21e3a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:13.475837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:13.476494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.476767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.477029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] PATH_INFO: `/v3/projects/572f6e477d1e4c93997a2405f40e526e/users/1dd3afc910a34fa5b0a8822d748200f0/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.478934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] RBAC: Authorizing `identity:create_grant(project_id=572f6e477d1e4c93997a2405f40e526e, user_id=1dd3afc910a34fa5b0a8822d748200f0, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:13.481936 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:13.492196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1325/2644] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:13 2022] PUT /identity/v3/projects/572f6e477d1e4c93997a2405f40e526e/users/1dd3afc910a34fa5b0a8822d748200f0/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:13.503364 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de3c80-0141-4f9f-8d2f-46207fbb4083 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.503652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de3c80-0141-4f9f-8d2f-46207fbb4083 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.504018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3de3c80-0141-4f9f-8d2f-46207fbb4083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.534707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a3de3c80-0141-4f9f-8d2f-46207fbb4083 None None] MFA Rules not processed for user `1dd3afc910a34fa5b0a8822d748200f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:13.535847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a3de3c80-0141-4f9f-8d2f-46207fbb4083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:13.605574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1320/2645] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:13 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:13.616399 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:13.618199 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:13.637823 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:13.676734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ef1b16-e9f7-47bf-85de-f0485b828c51 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:13.680362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901a5c27-b7bd-44cb-a7e2-61744b0c4b54 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:13.680952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901a5c27-b7bd-44cb-a7e2-61744b0c4b54 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:13.681228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901a5c27-b7bd-44cb-a7e2-61744b0c4b54 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:13.681496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901a5c27-b7bd-44cb-a7e2-61744b0c4b54 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:13.690395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901a5c27-b7bd-44cb-a7e2-61744b0c4b54 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:13.691801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901a5c27-b7bd-44cb-a7e2-61744b0c4b54 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:13.701565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1326/2646] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:13 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:14.072270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3de3c80-0141-4f9f-8d2f-46207fbb4083 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:14.110924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:14.134801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3de3c80-0141-4f9f-8d2f-46207fbb4083 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:14.139362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:14.140006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:14.140288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:14.140558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] PATH_INFO: `/v3/users/e8b50f6f3bad43ad9262aa34c2399321` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:14.141742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8b50f6f3bad43ad9262aa34c2399321)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:14.143700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:14.193555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8b50f6f3bad43ad9262aa34c2399321'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.194207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8b50f6f3bad43ad9262aa34c2399321'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.201705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8b50f6f3bad43ad9262aa34c2399321'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.202362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8b50f6f3bad43ad9262aa34c2399321'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.214985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8b50f6f3bad43ad9262aa34c2399321'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.231192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1321/2647] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:14 2022] DELETE /identity/v3/users/e8b50f6f3bad43ad9262aa34c2399321 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:14.236496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:14.236802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:14.237156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:14.258898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:14.259953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:14.318622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1327/2648] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:14.328852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:14.330276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:14.387962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ec8482-b5c3-4dcd-a7a5-34d98a4341e1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:14.391617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:14.392238 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:14.392514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:14.392777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:14.402266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:14.403665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:14.413909 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1322/2649] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:14.681180 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:14.682766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:14.699967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4638b0-40ad-4d80-a302-10e70012e130 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:14.704937 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:14.705572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:14.705857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:14.706198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] PATH_INFO: `/v3/projects/2e81b4600f474d0e9d8bc2f842ebc39e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:14.707252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e81b4600f474d0e9d8bc2f842ebc39e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:14.709143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:14.757527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e81b4600f474d0e9d8bc2f842ebc39e', 'request_id': 'req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.764353 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e81b4600f474d0e9d8bc2f842ebc39e', 'request_id': 'req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.769854 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e81b4600f474d0e9d8bc2f842ebc39e', 'request_id': 'req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.782463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] The token cache is being invalidate because project 2e81b4600f474d0e9d8bc2f842ebc39e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:14.782746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.783351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.784852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1328/2650] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:14 2022] DELETE /identity/v3/projects/2e81b4600f474d0e9d8bc2f842ebc39e => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:14.788872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:14.798898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fddd15-02e2-4792-a7cc-0ae88dcd3f24 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:14.852258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:14.859413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:14.860188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:14.860694 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:14.861195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] PATH_INFO: `/v3/users/1dd3afc910a34fa5b0a8822d748200f0` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:14.862499 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dd3afc910a34fa5b0a8822d748200f0)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:14.864529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:14.910281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dd3afc910a34fa5b0a8822d748200f0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.910919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dd3afc910a34fa5b0a8822d748200f0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.918964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dd3afc910a34fa5b0a8822d748200f0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.919585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dd3afc910a34fa5b0a8822d748200f0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.931151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-04dbdb89-0215-479e-9fee-fc8ebb7942d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dd3afc910a34fa5b0a8822d748200f0'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:14.944496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1323/2651] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:14 2022] DELETE /identity/v3/users/1dd3afc910a34fa5b0a8822d748200f0 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:15.214445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.215876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:15.230131 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb8a3c5-8247-4764-abf8-b2ccd7641c96 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.272119 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:15.291715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.292379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.292651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.292917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] PATH_INFO: `/v3/projects/572f6e477d1e4c93997a2405f40e526e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.294037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=572f6e477d1e4c93997a2405f40e526e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:15.295957 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:15.347544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '572f6e477d1e4c93997a2405f40e526e', 'request_id': 'req-61e95380-c00e-4373-9667-3e86573b3d9c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:15.354428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '572f6e477d1e4c93997a2405f40e526e', 'request_id': 'req-61e95380-c00e-4373-9667-3e86573b3d9c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:15.361113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '572f6e477d1e4c93997a2405f40e526e', 'request_id': 'req-61e95380-c00e-4373-9667-3e86573b3d9c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:15.374862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] The token cache is being invalidate because project 572f6e477d1e4c93997a2405f40e526e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:15.375161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:15.379879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-61e95380-c00e-4373-9667-3e86573b3d9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:15.381367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1329/2652] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:15 2022] DELETE /identity/v3/projects/572f6e477d1e4c93997a2405f40e526e => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:15.386591 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e46faf-06e1-4a25-b19a-2f9d4323f48e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.387063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e46faf-06e1-4a25-b19a-2f9d4323f48e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.387176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e46faf-06e1-4a25-b19a-2f9d4323f48e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.408755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-f4e46faf-06e1-4a25-b19a-2f9d4323f48e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:15.409891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f4e46faf-06e1-4a25-b19a-2f9d4323f48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:15.468587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1324/2653] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:15.476396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a0747-13a4-40f9-81e2-1566be6d90c1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.476683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a0747-13a4-40f9-81e2-1566be6d90c1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.476954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83a0747-13a4-40f9-81e2-1566be6d90c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.497071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a83a0747-13a4-40f9-81e2-1566be6d90c1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:15.498264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a83a0747-13a4-40f9-81e2-1566be6d90c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:15.506351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1330/2654] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:15.511302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e46faf-06e1-4a25-b19a-2f9d4323f48e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.521908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e46faf-06e1-4a25-b19a-2f9d4323f48e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.526826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.527416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.527718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.528017 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.528654 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:03:15.528957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:15.530551 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:15.543080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1325/2655] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:03:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:03:15.547746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a0747-13a4-40f9-81e2-1566be6d90c1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.557214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83a0747-13a4-40f9-81e2-1566be6d90c1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.562747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.563410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.563712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.564041 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.564672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:15.566554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:15.595143 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1331/2656] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:15 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:03:15.599345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.609426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcbf374-769d-4ba9-9119-c4a3a247fcdb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.665986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.666500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.666893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.667163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.667800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:15.669724 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:15.679630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:15.692123 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1326/2657] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:15 2022] POST /identity/v3/users => generated 286 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:03:15.696047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.707135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5e35de-4425-4653-a1ec-d67176f84716 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.711710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.712354 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.712629 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.712895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.713526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:15.715483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:15.725201 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1332/2658] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:15.729054 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.739730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1485a9e8-62d7-469c-9778-95f40188a4b8 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.744359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c281cd-7e2d-4003-89be-3f03a73d3f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.746113 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c281cd-7e2d-4003-89be-3f03a73d3f43 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.746389 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c281cd-7e2d-4003-89be-3f03a73d3f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.746655 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c281cd-7e2d-4003-89be-3f03a73d3f43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.747288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c281cd-7e2d-4003-89be-3f03a73d3f43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:15.748913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c281cd-7e2d-4003-89be-3f03a73d3f43 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:15.759587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1327/2659] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:15.764541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.774342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21133537-e1c9-4e8a-a9ba-c08656a8f038 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.779331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.779960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.780256 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.780526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] PATH_INFO: `/v3/projects/9506553a97aa4a81acaabf3daaed1a1f/users/23887511babe466fa4b78c3e2c1ba5e1/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.795356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9506553a97aa4a81acaabf3daaed1a1f, user_id=23887511babe466fa4b78c3e2c1ba5e1, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:15.798075 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:15.813033 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1333/2660] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:15 2022] PUT /identity/v3/projects/9506553a97aa4a81acaabf3daaed1a1f/users/23887511babe466fa4b78c3e2c1ba5e1/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:15.823106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dee03f9-e685-48e1-aab2-772f403d69c6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.823399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dee03f9-e685-48e1-aab2-772f403d69c6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.823672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dee03f9-e685-48e1-aab2-772f403d69c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:15.850410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-3dee03f9-e685-48e1-aab2-772f403d69c6 None None] MFA Rules not processed for user `23887511babe466fa4b78c3e2c1ba5e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:15.851607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3dee03f9-e685-48e1-aab2-772f403d69c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:15.921334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1328/2661] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:15 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:15.933876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:15.935522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:15.994195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65970f6b-5918-4394-83f1-a094557dcb8a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:15.998545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:15.999153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:15.999435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:15.999734 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:16.008322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:16.009741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:16.018489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1334/2662] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:15 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 02:03:25.917237 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=1,keystone:SELECT=39 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:26.014521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=85,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:39.577299 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b01b91-8ea9-4d36-b57d-fce80953df7a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:39.577821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b01b91-8ea9-4d36-b57d-fce80953df7a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:39.577934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b01b91-8ea9-4d36-b57d-fce80953df7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:39.596960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b9b01b91-8ea9-4d36-b57d-fce80953df7a None None] MFA Rules not processed for user `a14f57dd673f480c9984491a920fcacd`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:39.598150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b9b01b91-8ea9-4d36-b57d-fce80953df7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:39.653158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1329/2663] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:39 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:49.646450 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=25 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:51.849060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:51.850543 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:51.909530 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d6d08c-db61-4758-82db-3f97a24ee002 service glance] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:51.913126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f45a936-ec65-4de8-aa51-45c34e779d01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:51.913713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f45a936-ec65-4de8-aa51-45c34e779d01 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:51.913991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f45a936-ec65-4de8-aa51-45c34e779d01 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:51.914259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f45a936-ec65-4de8-aa51-45c34e779d01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:51.916015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4f45a936-ec65-4de8-aa51-45c34e779d01 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:52.105408 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f45a936-ec65-4de8-aa51-45c34e779d01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:52.106832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f45a936-ec65-4de8-aa51-45c34e779d01 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:52.118096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1335/2664] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:57.476838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b01b91-8ea9-4d36-b57d-fce80953df7a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:57.539544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b01b91-8ea9-4d36-b57d-fce80953df7a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:57.543953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:57.544613 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:57.544875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:57.545146 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] PATH_INFO: `/v3/users/23887511babe466fa4b78c3e2c1ba5e1` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:57.546265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=23887511babe466fa4b78c3e2c1ba5e1)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:57.548985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:57.593922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23887511babe466fa4b78c3e2c1ba5e1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:57.594574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23887511babe466fa4b78c3e2c1ba5e1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:57.601449 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23887511babe466fa4b78c3e2c1ba5e1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:57.602071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23887511babe466fa4b78c3e2c1ba5e1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:57.616585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23887511babe466fa4b78c3e2c1ba5e1'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:57.629728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1330/2665] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:03:57 2022] DELETE /identity/v3/users/23887511babe466fa4b78c3e2c1ba5e1 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:57.634331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:57.634814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:57.634934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:57.653435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:57.654616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:57.719070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1336/2666] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:57.728893 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:57.730287 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:57.789422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10d1173-d0bc-43ca-ae03-3a3cb28739a2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:57.793140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5501d64b-a341-4bed-84d1-664ba0112ee8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:57.793687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5501d64b-a341-4bed-84d1-664ba0112ee8 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:57.793952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5501d64b-a341-4bed-84d1-664ba0112ee8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:57.794218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5501d64b-a341-4bed-84d1-664ba0112ee8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:57.803848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5501d64b-a341-4bed-84d1-664ba0112ee8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:57.805360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5501d64b-a341-4bed-84d1-664ba0112ee8 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:57.815448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1331/2667] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:03:58.080486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.081878 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:58.100484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff867dea-18a5-42a8-b7b2-3140c5f89777 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.105974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.106773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.107059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.107331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] PATH_INFO: `/v3/projects/9506553a97aa4a81acaabf3daaed1a1f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.108484 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9506553a97aa4a81acaabf3daaed1a1f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.110555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.150157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:58.160366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9506553a97aa4a81acaabf3daaed1a1f', 'request_id': 'req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:58.166536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9506553a97aa4a81acaabf3daaed1a1f', 'request_id': 'req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:58.172513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9506553a97aa4a81acaabf3daaed1a1f', 'request_id': 'req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:58.184518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] The token cache is being invalidate because project 9506553a97aa4a81acaabf3daaed1a1f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:03:58.184802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:58.185400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-fe8f3525-d3c1-4cff-b389-1b0a05b1ecf0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:03:58.186847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1337/2668] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:03:58 2022] DELETE /identity/v3/projects/9506553a97aa4a81acaabf3daaed1a1f => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:03:58.192435 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d491f46-3032-4561-a820-f5a61f14757f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.192717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d491f46-3032-4561-a820-f5a61f14757f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.192986 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d491f46-3032-4561-a820-f5a61f14757f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.212901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5d491f46-3032-4561-a820-f5a61f14757f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:58.214078 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5d491f46-3032-4561-a820-f5a61f14757f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:58.230594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:03:58.277201 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1332/2669] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:58.285327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7801cf5c-ae24-4b99-9b30-618676f719ba None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.285616 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7801cf5c-ae24-4b99-9b30-618676f719ba None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.285897 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7801cf5c-ae24-4b99-9b30-618676f719ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.306443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-7801cf5c-ae24-4b99-9b30-618676f719ba None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:58.307637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7801cf5c-ae24-4b99-9b30-618676f719ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:58.316037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1338/2670] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:58.320246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d491f46-3032-4561-a820-f5a61f14757f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.330800 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d491f46-3032-4561-a820-f5a61f14757f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.335383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.336062 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.336340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.336607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.337244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:03:58.337539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.339102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.350946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1333/2671] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:03:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:03:58.355230 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7801cf5c-ae24-4b99-9b30-618676f719ba None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.364487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7801cf5c-ae24-4b99-9b30-618676f719ba None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.369108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.369694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.369970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.370239 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.370857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.372760 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.409903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1339/2672] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:03:58 2022] POST /identity/v3/projects => generated 399 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:03:58.414039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.424081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb6ef2e-ccf4-42c6-a80a-12c80cb5a2cc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.479776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.480556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.480837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.481107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.481728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.483731 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.505963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1334/2673] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:03:58 2022] POST /identity/v3/users => generated 300 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:03:58.510014 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.520146 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd01e993-de1e-47a0-bcca-182766b8e610 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.524882 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.525468 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.525749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.526015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.526649 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.528445 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.538269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1340/2674] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:58.542100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.552485 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eb50cb-0880-4195-ae93-57e2f004405c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.557172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586d80d2-3841-49ab-91e5-add30d99ec08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.557775 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d80d2-3841-49ab-91e5-add30d99ec08 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.558051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d80d2-3841-49ab-91e5-add30d99ec08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.558317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586d80d2-3841-49ab-91e5-add30d99ec08 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.558969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586d80d2-3841-49ab-91e5-add30d99ec08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.560616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586d80d2-3841-49ab-91e5-add30d99ec08 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.571596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1335/2675] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:03:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:03:58.575824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.585553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7daaee5b-7f3a-4389-99ca-56528d479082 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.590066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.590705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.590980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.591275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] PATH_INFO: `/v3/projects/b67d73cfbfaf479d8860323f6eafef3c/users/c30899c5383649c39a6a5ed9bb84d6b5/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.605912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b67d73cfbfaf479d8860323f6eafef3c, user_id=c30899c5383649c39a6a5ed9bb84d6b5, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.608551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.621741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1341/2676] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:03:58 2022] PUT /identity/v3/projects/b67d73cfbfaf479d8860323f6eafef3c/users/c30899c5383649c39a6a5ed9bb84d6b5/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:03:58.632751 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.633049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.633424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.660335 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] MFA Rules not processed for user `c30899c5383649c39a6a5ed9bb84d6b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:03:58.661550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:58.724646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1336/2677] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:03:58 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:03:58.736702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:03:58.738377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:03:58.794198 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fb7655-e9c5-44a1-b0d1-79830e88c9d4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:03:58.798588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:03:58.799179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:03:58.799453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:03:58.799717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:03:58.808347 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:03:58.809770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:03:58.819624 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1342/2678] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:03:58 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:01.862938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:01.864465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:01.884939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:01.923149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a2aff5-ae7e-4b14-a57d-34cc11528080 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:01.926677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46877838-272f-4269-ac3a-00d5c908a29d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:01.927332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46877838-272f-4269-ac3a-00d5c908a29d service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:01.927606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46877838-272f-4269-ac3a-00d5c908a29d service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:01.927871 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46877838-272f-4269-ac3a-00d5c908a29d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:01.929596 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-46877838-272f-4269-ac3a-00d5c908a29d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:01.997692 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46877838-272f-4269-ac3a-00d5c908a29d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:01.999111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46877838-272f-4269-ac3a-00d5c908a29d service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:02.010539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1337/2679] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:08.815554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=68,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:12.005846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=46 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:12.273625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:12.344437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c461b-6101-4458-bff9-cadfa34caa2f service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:12.349004 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:12.349871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:12.350197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:12.350395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] PATH_INFO: `/v3/users/c30899c5383649c39a6a5ed9bb84d6b5` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:12.352514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c30899c5383649c39a6a5ed9bb84d6b5)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:12.354480 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:12.419868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c30899c5383649c39a6a5ed9bb84d6b5'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:12.420564 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c30899c5383649c39a6a5ed9bb84d6b5'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:12.422000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c30899c5383649c39a6a5ed9bb84d6b5'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:12.429531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c30899c5383649c39a6a5ed9bb84d6b5'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:12.444964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c30899c5383649c39a6a5ed9bb84d6b5'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:12.458433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1343/2680] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:04:12 2022] DELETE /identity/v3/users/c30899c5383649c39a6a5ed9bb84d6b5 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:12.462863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:12.463162 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:12.463420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:12.483195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:12.485467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:12.545221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1338/2681] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:12.555780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:12.557303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:12.623707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe8ee38-3b89-4e85-bd3c-b5706fb68cc4 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:12.629204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2863f53d-ace1-4a2d-877b-d4f3ec102d42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:12.629961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863f53d-ace1-4a2d-877b-d4f3ec102d42 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:12.630242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863f53d-ace1-4a2d-877b-d4f3ec102d42 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:12.630508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2863f53d-ace1-4a2d-877b-d4f3ec102d42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:12.640379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2863f53d-ace1-4a2d-877b-d4f3ec102d42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:12.641808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2863f53d-ace1-4a2d-877b-d4f3ec102d42 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:12.651294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1344/2682] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:13.075690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.077742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:13.106774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038e2a4e-666d-4b95-8d6a-c145c6a4af5c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.113329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.113969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.114247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.114546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] PATH_INFO: `/v3/projects/b67d73cfbfaf479d8860323f6eafef3c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.115673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=b67d73cfbfaf479d8860323f6eafef3c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.117727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.166578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b67d73cfbfaf479d8860323f6eafef3c', 'request_id': 'req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:13.176045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b67d73cfbfaf479d8860323f6eafef3c', 'request_id': 'req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:13.182137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b67d73cfbfaf479d8860323f6eafef3c', 'request_id': 'req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:13.200242 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] The token cache is being invalidate because project b67d73cfbfaf479d8860323f6eafef3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:04:13.200531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:13.201155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d87834c4-0f69-49cb-ac6b-03d926c2a9ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:13.203372 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1339/2683] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:04:13 2022] DELETE /identity/v3/projects/b67d73cfbfaf479d8860323f6eafef3c => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:13.208234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8939dd6-8e02-4fab-b002-5216657f79fc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.208527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8939dd6-8e02-4fab-b002-5216657f79fc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.208809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8939dd6-8e02-4fab-b002-5216657f79fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.230027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a8939dd6-8e02-4fab-b002-5216657f79fc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:13.231355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a8939dd6-8e02-4fab-b002-5216657f79fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:13.252314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:13.299221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1345/2684] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:13.307897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e98a678-282f-4d88-8b12-5ad4d45b962f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.308221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e98a678-282f-4d88-8b12-5ad4d45b962f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.308518 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e98a678-282f-4d88-8b12-5ad4d45b962f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.329247 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5e98a678-282f-4d88-8b12-5ad4d45b962f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:13.330440 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5e98a678-282f-4d88-8b12-5ad4d45b962f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:13.338360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1340/2685] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:13.342646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8939dd6-8e02-4fab-b002-5216657f79fc None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.352448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8939dd6-8e02-4fab-b002-5216657f79fc None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.356926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.357508 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.357801 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.358050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.358906 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:04:13.359307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.360915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.372219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1346/2686] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:04:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 02:04:13.377486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e98a678-282f-4d88-8b12-5ad4d45b962f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.386723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e98a678-282f-4d88-8b12-5ad4d45b962f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.392036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.392623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.392903 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.393174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.393807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.395706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.425855 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1341/2687] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:04:13 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:04:13.427268 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.437887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b678ad-beba-4bcc-a503-81861ee27e19 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.493079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.493682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.493965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.494233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.494859 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.496803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.519638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1347/2688] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:04:13 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:04:13.523859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.534203 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f12cc9a-dcde-4851-b761-351a75a36880 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.539086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c102487-4001-416c-8647-732330676c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.539683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c102487-4001-416c-8647-732330676c44 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.540049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c102487-4001-416c-8647-732330676c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.540324 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c102487-4001-416c-8647-732330676c44 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.540982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c102487-4001-416c-8647-732330676c44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.542565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c102487-4001-416c-8647-732330676c44 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.553112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1342/2689] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:04:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:04:13.556958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.567441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af550ed9-6e21-4ccc-a6cd-2222307391a5 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.572058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0591795b-e415-49c2-920b-4b6374878e87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.572672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0591795b-e415-49c2-920b-4b6374878e87 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.572929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0591795b-e415-49c2-920b-4b6374878e87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.573216 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0591795b-e415-49c2-920b-4b6374878e87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.573858 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0591795b-e415-49c2-920b-4b6374878e87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.575424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0591795b-e415-49c2-920b-4b6374878e87 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.586159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1348/2690] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:04:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:04:13.590285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c102487-4001-416c-8647-732330676c44 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.600427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c102487-4001-416c-8647-732330676c44 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.605399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.606155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.606479 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.606797 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] PATH_INFO: `/v3/projects/5318d2681e9d4fd2adc29977165e0a9b/users/e961b9e63ece47e8934b7e4c7b5c46ff/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.622946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5318d2681e9d4fd2adc29977165e0a9b, user_id=e961b9e63ece47e8934b7e4c7b5c46ff, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.626519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.639045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1343/2691] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:04:13 2022] PUT /identity/v3/projects/5318d2681e9d4fd2adc29977165e0a9b/users/e961b9e63ece47e8934b7e4c7b5c46ff/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:13.644052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf454ea7-d654-408c-96c9-35d8408bb6dd None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.644363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf454ea7-d654-408c-96c9-35d8408bb6dd None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.644637 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf454ea7-d654-408c-96c9-35d8408bb6dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.663057 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-bf454ea7-d654-408c-96c9-35d8408bb6dd None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:13.664406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bf454ea7-d654-408c-96c9-35d8408bb6dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:13.723107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1349/2692] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:13.732679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:13.735098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:13.757292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:13.793748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95572233-b801-4989-9f07-3381176481b4 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:13.798263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:13.798857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:13.799132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:13.799398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:13.808764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:13.810343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:13.820650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1344/2693] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:18.090853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9ca355-0115-45ac-ad60-863f6cb31bee None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:18.091343 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9ca355-0115-45ac-ad60-863f6cb31bee None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:18.091440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9ca355-0115-45ac-ad60-863f6cb31bee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:18.121300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-ee9ca355-0115-45ac-ad60-863f6cb31bee None None] MFA Rules not processed for user `e961b9e63ece47e8934b7e4c7b5c46ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:18.122551 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-ee9ca355-0115-45ac-ad60-863f6cb31bee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:18.141949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:18.204171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1350/2694] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:18 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:18.208273 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:18.221063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b524ba01-b207-4922-a23b-c0140d89b8e9 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:18.227274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:18.228784 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:18.229070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:18.229343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:18.229999 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:18.232428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:18.265218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1345/2695] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:04:18 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:04:18.267006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9ca355-0115-45ac-ad60-863f6cb31bee None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:18.278191 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9ca355-0115-45ac-ad60-863f6cb31bee None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:18.343458 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:18.344646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:18.344646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:18.344980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:18.345549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:18.347729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:18.371176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1351/2696] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:04:18 2022] POST /identity/v3/users => generated 298 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:04:18.376107 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:18.386647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd223a5-c7a3-4559-a5f7-5ab8057c330c admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:18.392254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:18.392970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:18.393255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:18.393525 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:18.394176 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:18.395910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:18.407671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1346/2697] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:04:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:04:18.412154 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:18.424582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0902a2a1-859e-42ca-a4bc-c3965a9a820f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:18.429425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:18.430101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:18.430385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:18.430655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] PATH_INFO: `/v3/projects/e60745c3d9fa4555be6fb8e71f2dbe6a/users/3909c2287b3446f980075790b09d3f06/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:18.448688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e60745c3d9fa4555be6fb8e71f2dbe6a, user_id=3909c2287b3446f980075790b09d3f06, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:18.451339 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:18.465790 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1352/2698] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:04:18 2022] PUT /identity/v3/projects/e60745c3d9fa4555be6fb8e71f2dbe6a/users/3909c2287b3446f980075790b09d3f06/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:04:18.470108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:18.480921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd2ce29-5372-4a46-ae03-53e5b1693f9e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:18.548960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:18.549750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:18.550094 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:18.550368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:18.551012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:18.552730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:18.564341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1347/2699] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:04:18 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:04:18.568815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:18.580839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb8bb2d-4676-445c-b33e-d4d0e6f1fd5f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:18.586478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4de34b-027a-4ddf-93b7-7f21bd9f2d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:18.587123 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4de34b-027a-4ddf-93b7-7f21bd9f2d27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:18.587401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4de34b-027a-4ddf-93b7-7f21bd9f2d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:18.587671 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4de34b-027a-4ddf-93b7-7f21bd9f2d27 admin admin] PATH_INFO: `/v3/projects/e60745c3d9fa4555be6fb8e71f2dbe6a/users/3909c2287b3446f980075790b09d3f06/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:18.589625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4de34b-027a-4ddf-93b7-7f21bd9f2d27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e60745c3d9fa4555be6fb8e71f2dbe6a, user_id=3909c2287b3446f980075790b09d3f06, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:18.593475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4de34b-027a-4ddf-93b7-7f21bd9f2d27 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:18.609644 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1353/2700] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:04:18 2022] PUT /identity/v3/projects/e60745c3d9fa4555be6fb8e71f2dbe6a/users/3909c2287b3446f980075790b09d3f06/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:04:19.923077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:19.996903 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee6d317-43a5-48d1-ba27-5dd09f3e528b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:20.076010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:20.088973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:20.089696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:20.090190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:20.090671 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:20.093543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:20.167788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:20.169408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:20.181526 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1348/2701] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 02:04:24.290133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbbb1d6-1ccc-49e3-8013-7094f7df68a0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:24.291678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbbb1d6-1ccc-49e3-8013-7094f7df68a0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:24.292195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbbb1d6-1ccc-49e3-8013-7094f7df68a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:24.342053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-adbbb1d6-1ccc-49e3-8013-7094f7df68a0 None None] MFA Rules not processed for user `3909c2287b3446f980075790b09d3f06`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:24.344679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-adbbb1d6-1ccc-49e3-8013-7094f7df68a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:24.419814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:24.433738 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1354/2702] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:24 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:24.447252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:24.448698 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:24.572444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feacf7d4-789f-455f-9da3-b27018e0c422 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:24.578754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:24.584777 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:24.585304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:24.585646 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:24.698563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:24.700045 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:24.711539 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1349/2703] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:24 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:24.936680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:24.937177 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:24.937739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:25.027872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:25.029395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:25.119117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1355/2704] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 02:04:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:25.186715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:25.188857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:25.273730 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:25.323825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca51e496-3e09-4d42-ab60-47798b4000f8 service glance] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:25.332262 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:25.332889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:25.333354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:25.333484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:25.346327 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:25.347983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:25.358511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1350/2705] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 02:04:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 02:04:25.594124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:25.595089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:25.646680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33351605-d755-41a8-a46d-a9d2e8b4ec29 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:25.669220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:25.669871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:25.670148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:25.670474 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] PATH_INFO: `/v3/users/dd95ef263d15460f9cca5a7557cbee2e` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:25.716505 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd95ef263d15460f9cca5a7557cbee2e)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:25.719755 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:25.852094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd95ef263d15460f9cca5a7557cbee2e'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:25.853037 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd95ef263d15460f9cca5a7557cbee2e'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:25.853980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd95ef263d15460f9cca5a7557cbee2e'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:25.884271 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd95ef263d15460f9cca5a7557cbee2e'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:25.957495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-14bead67-3f6d-47b3-a298-1e5f75c7eba3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd95ef263d15460f9cca5a7557cbee2e'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:25.993156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1356/2706] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:04:25 2022] DELETE /identity/v3/users/dd95ef263d15460f9cca5a7557cbee2e => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:26.288185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:26.304632 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410aedfe-ac94-4cf4-8981-9046e38a0e3e service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:26.392627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:26.393278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:26.394304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:26.394584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] PATH_INFO: `/v3/projects/1377f60423c64a1786203deb55ae55ef` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:26.410430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] RBAC: Authorizing `identity:delete_project(project_id=1377f60423c64a1786203deb55ae55ef)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:26.412690 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:26.513645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1377f60423c64a1786203deb55ae55ef', 'request_id': 'req-3282176c-335c-4ed2-a3d1-872f848fb2be'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:26.533766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1377f60423c64a1786203deb55ae55ef', 'request_id': 'req-3282176c-335c-4ed2-a3d1-872f848fb2be'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:26.545910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1377f60423c64a1786203deb55ae55ef', 'request_id': 'req-3282176c-335c-4ed2-a3d1-872f848fb2be'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:26.577441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] The token cache is being invalidate because project 1377f60423c64a1786203deb55ae55ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:04:26.577835 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:26.578461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-3282176c-335c-4ed2-a3d1-872f848fb2be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:26.584310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1351/2707] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:04:26 2022] DELETE /identity/v3/projects/1377f60423c64a1786203deb55ae55ef => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:26.587070 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa2b6c1-db95-4199-a292-e5c7219b9155 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:26.587390 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa2b6c1-db95-4199-a292-e5c7219b9155 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:26.587669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa2b6c1-db95-4199-a292-e5c7219b9155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:26.632473 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-2fa2b6c1-db95-4199-a292-e5c7219b9155 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:26.633872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2fa2b6c1-db95-4199-a292-e5c7219b9155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:26.667748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:26.743779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1357/2708] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:26.752437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd1b7f9-1ba0-4703-bdae-51551b66ff83 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:26.752743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd1b7f9-1ba0-4703-bdae-51551b66ff83 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:26.752990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd1b7f9-1ba0-4703-bdae-51551b66ff83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:26.775253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0bd1b7f9-1ba0-4703-bdae-51551b66ff83 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:26.776869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0bd1b7f9-1ba0-4703-bdae-51551b66ff83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:26.786781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1352/2709] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:26.791110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa2b6c1-db95-4199-a292-e5c7219b9155 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:26.806628 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa2b6c1-db95-4199-a292-e5c7219b9155 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:26.813874 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:26.814683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:26.814969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:26.815409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:26.816170 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:04:26.816638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:26.818385 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:26.835991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1358/2710] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:04:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:04:26.840223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd1b7f9-1ba0-4703-bdae-51551b66ff83 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:26.852252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd1b7f9-1ba0-4703-bdae-51551b66ff83 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:26.857113 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:26.858353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:26.858750 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:26.859022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:26.859658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:26.861630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:26.901134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1353/2711] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:04:26 2022] POST /identity/v3/projects => generated 383 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:04:26.908226 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:26.924442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d2766-0f63-48df-ba55-2476142dba03 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:27.039743 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:27.040723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:27.040886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:27.041169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:27.041792 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:27.044089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:27.088402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:27.088772 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1359/2712] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:04:26 2022] POST /identity/v3/users => generated 292 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:04:27.101746 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7496a9-d1e2-4314-9a1a-4dfcd8913dda admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:27.108886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:27.109625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:27.110004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:27.110375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:27.111224 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:27.113000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:27.118091 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:27.135394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:27.146140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1354/2713] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:04:27 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:04:27.148018 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec36e262-8052-4e09-9aa6-c41bcaf95a35 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:27.157272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda4b9e3-6a4b-4849-82d8-2beedc11fef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:27.158011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4b9e3-6a4b-4849-82d8-2beedc11fef2 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:27.158456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4b9e3-6a4b-4849-82d8-2beedc11fef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:27.158947 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda4b9e3-6a4b-4849-82d8-2beedc11fef2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:27.159702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda4b9e3-6a4b-4849-82d8-2beedc11fef2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:27.161428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda4b9e3-6a4b-4849-82d8-2beedc11fef2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:27.188526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1360/2714] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:04:27 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:04:27.192210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:27.205053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a55a776-56e2-4a20-8c40-7d7d6abdd583 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:27.212620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:27.214187 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:27.214758 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:27.215170 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] PATH_INFO: `/v3/projects/663bde66131d47159ffa09ad33971895/users/86bc1b5163f4453db4469fef90f27344/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:27.234051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=663bde66131d47159ffa09ad33971895, user_id=86bc1b5163f4453db4469fef90f27344, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:27.237951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:27.278679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1355/2715] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:04:27 2022] PUT /identity/v3/projects/663bde66131d47159ffa09ad33971895/users/86bc1b5163f4453db4469fef90f27344/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:27.280774 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78deeef5-4834-4fa0-bcfb-bff4d7137f29 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:27.281093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78deeef5-4834-4fa0-bcfb-bff4d7137f29 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:27.281375 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78deeef5-4834-4fa0-bcfb-bff4d7137f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:27.315606 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-78deeef5-4834-4fa0-bcfb-bff4d7137f29 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:27.316890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-78deeef5-4834-4fa0-bcfb-bff4d7137f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:27.446304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1361/2716] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:04:27.456705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:27.459743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:27.586305 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4b6227-e7c3-44ef-b717-725baa54af1e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:27.597923 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:27.599223 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:27.599848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:27.600486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:27.613997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:27.618483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:27.633035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1356/2717] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:33.457666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64002338-e527-44dd-8808-b48567abc665 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:33.458184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64002338-e527-44dd-8808-b48567abc665 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:33.458184 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64002338-e527-44dd-8808-b48567abc665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:33.519314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-64002338-e527-44dd-8808-b48567abc665 None None] MFA Rules not processed for user `86bc1b5163f4453db4469fef90f27344`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:33.520586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-64002338-e527-44dd-8808-b48567abc665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:33.555879 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:33.645233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1362/2718] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:33.677049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:33.679159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:33.887039 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f1cbd-94a2-4a7f-bce2-2238d25e06fa service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:33.904096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70475fd5-1a34-4a16-b8c9-d75fbd179679 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:33.905573 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70475fd5-1a34-4a16-b8c9-d75fbd179679 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:33.905816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70475fd5-1a34-4a16-b8c9-d75fbd179679 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:33.906069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70475fd5-1a34-4a16-b8c9-d75fbd179679 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:33.920021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70475fd5-1a34-4a16-b8c9-d75fbd179679 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:33.921727 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70475fd5-1a34-4a16-b8c9-d75fbd179679 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:33.938380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1357/2719] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 02:04:35.638844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e490686a-5c6f-4c1e-a540-0af411ae6df5 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:35.639395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e490686a-5c6f-4c1e-a540-0af411ae6df5 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:35.639852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e490686a-5c6f-4c1e-a540-0af411ae6df5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:35.660934 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-e490686a-5c6f-4c1e-a540-0af411ae6df5 None None] MFA Rules not processed for user `86bc1b5163f4453db4469fef90f27344`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:35.662280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e490686a-5c6f-4c1e-a540-0af411ae6df5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:35.684223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1363/2720] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:38.130122 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:38.130633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:38.130633 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:38.162464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] MFA Rules not processed for user `e961b9e63ece47e8934b7e4c7b5c46ff`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:38.163676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:38.262710 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1358/2721] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:04:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:43.875609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:43.876171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:43.876171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:43.895111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:04:43.896346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:43.953992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1364/2722] 10.222.0.42 () {58 vars in 1009 bytes} [Fri Sep 16 02:04:43 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:04:43.969351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:43.970806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:43.980398 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:44.044685 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efb1818-37fc-4e5c-acc8-4c04d3ba0777 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:44.051402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:44.052292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:44.052597 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:44.052881 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:44.062968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:44.064614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:44.073043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1359/2723] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 02:04:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 02:04:53.781377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:53.782821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:53.797270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49953c7a-10cf-44f2-a87b-411fc7837c78 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:53.801720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:53.802346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:53.802622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:53.802890 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] PATH_INFO: `/v3/users/a14f57dd673f480c9984491a920fcacd` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:53.804066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a14f57dd673f480c9984491a920fcacd)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:53.806001 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:53.851432 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a14f57dd673f480c9984491a920fcacd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:53.852114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a14f57dd673f480c9984491a920fcacd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:53.852714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a14f57dd673f480c9984491a920fcacd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:53.859582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a14f57dd673f480c9984491a920fcacd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:53.871082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a14f57dd673f480c9984491a920fcacd'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:53.883737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1365/2724] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:04:53 2022] DELETE /identity/v3/users/a14f57dd673f480c9984491a920fcacd => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:54.068205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:54.070755 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=29 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:54.085143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34013d0f-76c7-4d84-a3b3-864b5a4ffa96 service swift] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:54.151802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:54.152469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:54.152789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:54.153067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] PATH_INFO: `/v3/projects/0b14dad85f52497c9819f0f3bb397f18` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:54.154208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b14dad85f52497c9819f0f3bb397f18)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:54.156245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:54.204874 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b14dad85f52497c9819f0f3bb397f18', 'request_id': 'req-cc03249f-770e-473e-8795-2f2d7dbbd40a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:54.211138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b14dad85f52497c9819f0f3bb397f18', 'request_id': 'req-cc03249f-770e-473e-8795-2f2d7dbbd40a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:54.216641 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b14dad85f52497c9819f0f3bb397f18', 'request_id': 'req-cc03249f-770e-473e-8795-2f2d7dbbd40a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:54.229524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] The token cache is being invalidate because project 0b14dad85f52497c9819f0f3bb397f18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:04:54.229810 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:54.230424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:04:54.231897 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1360/2725] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:04:54 2022] DELETE /identity/v3/projects/0b14dad85f52497c9819f0f3bb397f18 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:04:54.636866 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:54.638395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:54.641344 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:54.642768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:54.679448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:54.702317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b44c6ac-df47-4288-a313-9a9fa5aa4316 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:54.703314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc03249f-770e-473e-8795-2f2d7dbbd40a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:54.706236 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:54.706969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:54.707245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:54.707555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:54.707858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:54.708034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:54.708149 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:54.708556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:54.709852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:54.710929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:54.719837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:54.721719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:54.724611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:54.726322 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:54.730200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1361/2726] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:54.738918 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1366/2727] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:04:55.966709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:04:55.971801 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:56.017484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:04:56.062739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d554a454-c836-472f-83c8-963cb717bfe3 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:04:56.068917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:04:56.069602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:04:56.069877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:04:56.070192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:04:56.072102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:04:56.086477 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:04:56.088095 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:04:56.099653 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1362/2728] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:04:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:05:03.739150 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:05:03.740651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:05:03.756843 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1966d12d-0525-45de-a432-4e891626db23 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:05:03.763113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d4dc7d-2f2e-4f61-9453-65e8681af201 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:05:03.763904 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d4dc7d-2f2e-4f61-9453-65e8681af201 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:05:03.764238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d4dc7d-2f2e-4f61-9453-65e8681af201 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:05:03.765054 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d4dc7d-2f2e-4f61-9453-65e8681af201 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:05:03.766898 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c9d4dc7d-2f2e-4f61-9453-65e8681af201 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:05:03.779739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d4dc7d-2f2e-4f61-9453-65e8681af201 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:05:03.782925 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d4dc7d-2f2e-4f61-9453-65e8681af201 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:05:03.794151 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1367/2729] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:05:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:05:06.094672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=19 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:05:13.791255 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=16 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:05:16.587476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:05:16.597456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842e69df-3503-4a8e-b955-4d84bd14a23d service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:05:16.601013 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:05:16.601593 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:05:16.601865 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:05:16.602127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:05:16.603805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:05:16.785667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:05:16.787072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:05:16.797184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1363/2730] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:05:16 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:05:26.792482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=30 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:06:35.836554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:06:35.837083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:06:35.837083 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:06:35.855750 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] MFA Rules not processed for user `86bc1b5163f4453db4469fef90f27344`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:06:35.856982 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:06:35.910917 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1368/2731] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:06:35 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:06:36.219524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:06:36.220994 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:06:36.232418 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ad980-4dd1-4ea3-93b4-6c37f042dec9 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:06:36.236031 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:06:36.236614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:06:36.236890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:06:36.237155 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:06:36.238812 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:06:36.248662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:06:36.250361 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:06:36.258399 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1364/2732] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:06:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:06:36.572536 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:06:36.574106 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:06:36.644252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90513d26-f689-4652-90f8-4c215fb2c269 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:06:36.649708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:06:36.650304 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:06:36.650586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:06:36.650865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:06:36.652600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:06:36.664381 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:06:36.665937 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:06:36.675672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1369/2733] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:06:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:06:46.254662 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:06:46.671542 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=57 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:06:47.168466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:06:47.181830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2977dba6-2dc5-4505-820f-0d558bb13e0c service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:06:47.187547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae162350-fc73-4a2f-8d0a-275213100d96 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:06:47.188159 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae162350-fc73-4a2f-8d0a-275213100d96 service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:06:47.188430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae162350-fc73-4a2f-8d0a-275213100d96 service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:06:47.188691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae162350-fc73-4a2f-8d0a-275213100d96 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:06:47.190395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ae162350-fc73-4a2f-8d0a-275213100d96 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:06:47.203464 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae162350-fc73-4a2f-8d0a-275213100d96 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:06:47.205069 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae162350-fc73-4a2f-8d0a-275213100d96 service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:06:47.217012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1365/2734] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:06:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:06:57.212169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:07:11.763881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:11.774526 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04e58a2-ce60-4c2d-b811-7ef9d789fd91 service placement] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:11.779465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:11.780089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:11.780368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:11.780634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:11.782307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:11.793612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:11.795162 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:11.805998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1370/2735] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:07:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:07:21.801463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:07:44.022503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6c1dcf-53d1-41db-b525-407ef8ce58e0 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:44.022983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6c1dcf-53d1-41db-b525-407ef8ce58e0 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:44.023100 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6c1dcf-53d1-41db-b525-407ef8ce58e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:44.042420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-ad6c1dcf-53d1-41db-b525-407ef8ce58e0 None None] MFA Rules not processed for user `f58ec160611e446193b43582a8155284`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:07:44.043587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ad6c1dcf-53d1-41db-b525-407ef8ce58e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:44.097180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1366/2736] 10.222.0.42 () {58 vars in 967 bytes} [Fri Sep 16 02:07:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:07:44.138043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:44.139442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:44.207202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd731e58-1b23-4482-8303-edee24ac46ee service placement] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:44.212066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'af7a48729497436784b988c6291cfa5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:44.212659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:44.212937 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:44.213204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:44.221138 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:44.222753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:44.230719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1371/2737] 10.222.0.42 () {60 vars in 1384 bytes} [Fri Sep 16 02:07:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 16 02:07:50.514038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6c1dcf-53d1-41db-b525-407ef8ce58e0 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:50.531363 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6c1dcf-53d1-41db-b525-407ef8ce58e0 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:50.537658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:50.538468 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:50.538742 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:50.539051 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:50.540950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:50.553971 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:50.555549 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:50.567108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1367/2738] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:07:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:07:50.581775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:50.583560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:50.651851 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ddbf7f-1e81-462c-b193-becd0270652a service swift] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:50.656321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:50.656932 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:50.657205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:50.657470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] PATH_INFO: `/v3/users/e961b9e63ece47e8934b7e4c7b5c46ff` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:50.658560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e961b9e63ece47e8934b7e4c7b5c46ff)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:50.661450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:50.706211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e961b9e63ece47e8934b7e4c7b5c46ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:50.707396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e961b9e63ece47e8934b7e4c7b5c46ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:50.708100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e961b9e63ece47e8934b7e4c7b5c46ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:50.715448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e961b9e63ece47e8934b7e4c7b5c46ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:50.728259 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e961b9e63ece47e8934b7e4c7b5c46ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:50.740800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1372/2739] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:07:50 2022] DELETE /identity/v3/users/e961b9e63ece47e8934b7e4c7b5c46ff => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:07:50.914437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:50.924885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db8ed09-3f22-4f37-a183-ddd8fa992d75 service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:50.985351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:50.985968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:50.986244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:50.986510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] PATH_INFO: `/v3/projects/5318d2681e9d4fd2adc29977165e0a9b` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:50.987608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=5318d2681e9d4fd2adc29977165e0a9b)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:50.989566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.034504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5318d2681e9d4fd2adc29977165e0a9b', 'request_id': 'req-1fa3b53a-7676-4267-8255-5977ab9d76fc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.041177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5318d2681e9d4fd2adc29977165e0a9b', 'request_id': 'req-1fa3b53a-7676-4267-8255-5977ab9d76fc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.046545 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5318d2681e9d4fd2adc29977165e0a9b', 'request_id': 'req-1fa3b53a-7676-4267-8255-5977ab9d76fc'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.058468 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] The token cache is being invalidate because project 5318d2681e9d4fd2adc29977165e0a9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:07:51.058753 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.059366 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.060912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1368/2740] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:07:50 2022] DELETE /identity/v3/projects/5318d2681e9d4fd2adc29977165e0a9b => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:07:51.065084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:51.067485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:51.080914 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5869670-8ab4-42fc-a7ba-4cf633496760 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:51.086611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:07:51.141563 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:51.142188 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.142463 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.142846 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] PATH_INFO: `/v3/users/3909c2287b3446f980075790b09d3f06` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.143901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3909c2287b3446f980075790b09d3f06)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:51.146889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.193965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3909c2287b3446f980075790b09d3f06'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.194618 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3909c2287b3446f980075790b09d3f06'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.195214 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3909c2287b3446f980075790b09d3f06'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.202134 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3909c2287b3446f980075790b09d3f06'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.214771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-1ad80f44-9205-49bf-8146-7390baa7a762 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3909c2287b3446f980075790b09d3f06'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.226857 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1373/2741] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:07:51 2022] DELETE /identity/v3/users/3909c2287b3446f980075790b09d3f06 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:07:51.396180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:51.397716 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:51.413243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa3b53a-7676-4267-8255-5977ab9d76fc admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:51.435743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:07:51.485068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:51.485840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.486118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.486388 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] PATH_INFO: `/v3/projects/e60745c3d9fa4555be6fb8e71f2dbe6a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.487519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e60745c3d9fa4555be6fb8e71f2dbe6a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:51.490204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.537748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e60745c3d9fa4555be6fb8e71f2dbe6a', 'request_id': 'req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.544346 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e60745c3d9fa4555be6fb8e71f2dbe6a', 'request_id': 'req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.551351 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e60745c3d9fa4555be6fb8e71f2dbe6a', 'request_id': 'req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.564058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] The token cache is being invalidate because project e60745c3d9fa4555be6fb8e71f2dbe6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:07:51.564337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.564952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-712a1c3d-8b4f-4bd1-8095-2cf96530ab22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:07:51.567197 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1369/2742] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:07:51 2022] DELETE /identity/v3/projects/e60745c3d9fa4555be6fb8e71f2dbe6a => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:07:51.572344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c7355e-e7f5-4d07-885a-9f1628d9c106 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.572571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c7355e-e7f5-4d07-885a-9f1628d9c106 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.572842 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c7355e-e7f5-4d07-885a-9f1628d9c106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.593020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f5c7355e-e7f5-4d07-885a-9f1628d9c106 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:07:51.594223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f5c7355e-e7f5-4d07-885a-9f1628d9c106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:51.657522 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1374/2743] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:07:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:07:51.665866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25362f40-f74b-4c30-a2f5-a66384289557 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.666284 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25362f40-f74b-4c30-a2f5-a66384289557 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.666556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25362f40-f74b-4c30-a2f5-a66384289557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.687977 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-25362f40-f74b-4c30-a2f5-a66384289557 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:07:51.689483 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-25362f40-f74b-4c30-a2f5-a66384289557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:51.698840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1370/2744] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:07:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:07:51.703137 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c7355e-e7f5-4d07-885a-9f1628d9c106 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:51.713440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c7355e-e7f5-4d07-885a-9f1628d9c106 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:51.722496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:51.723097 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.724176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.724176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.724393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:07:51.724562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:51.726165 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.738997 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1375/2745] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:07:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:07:51.744274 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25362f40-f74b-4c30-a2f5-a66384289557 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:51.754776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25362f40-f74b-4c30-a2f5-a66384289557 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:51.760420 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:51.761018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.761408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.761679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.762303 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:51.764533 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.794852 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1371/2746] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:07:51 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:07:51.799185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:51.811404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd4d0029-b108-40c3-a738-24cf8078b757 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:51.828016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:07:51.878994 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:51.879714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.880072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.880554 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.881541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:51.883475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.906586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1376/2747] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:07:51 2022] POST /identity/v3/users => generated 290 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:07:51.914348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:51.924829 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9b65ba-e655-4415-941a-ef1bbb44b649 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:51.930401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:51.930996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.931270 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.931531 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.932188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:51.934046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.945010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1372/2748] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:07:51 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:07:51.949587 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:51.961047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df638e51-a190-46ca-955d-3142c1103085 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:51.965933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c768d3f-d8cb-440e-b033-03c843d2f704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:51.966521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c768d3f-d8cb-440e-b033-03c843d2f704 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:51.966798 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c768d3f-d8cb-440e-b033-03c843d2f704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:51.967065 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c768d3f-d8cb-440e-b033-03c843d2f704 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:51.967696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c768d3f-d8cb-440e-b033-03c843d2f704 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:51.969325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c768d3f-d8cb-440e-b033-03c843d2f704 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:51.985176 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1377/2749] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:07:51 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:07:51.990622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:52.001042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c20cfa-8128-4673-9c79-4489c6b149da admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:52.005696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:52.006329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:52.006602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:52.006866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] PATH_INFO: `/v3/projects/9cfcd2376b1c4398befa49b423aaae26/users/772659a1932d45dea2c5120c4497314b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:52.026432 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cfcd2376b1c4398befa49b423aaae26, user_id=772659a1932d45dea2c5120c4497314b, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:52.029135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:52.047683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1373/2750] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:07:51 2022] PUT /identity/v3/projects/9cfcd2376b1c4398befa49b423aaae26/users/772659a1932d45dea2c5120c4497314b/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:07:52.053528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac63bc-9c7f-4245-bdb7-6b60aedefd3f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:52.053814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac63bc-9c7f-4245-bdb7-6b60aedefd3f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:52.054133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac63bc-9c7f-4245-bdb7-6b60aedefd3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:52.076118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8dac63bc-9c7f-4245-bdb7-6b60aedefd3f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:07:52.077368 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8dac63bc-9c7f-4245-bdb7-6b60aedefd3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:52.149250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1378/2751] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:07:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:07:52.160317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:52.162951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:52.223854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:07:52.228819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfa5c36-d9f6-4e1b-842f-309a8fb9a8f9 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:52.234204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:52.235086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:52.235816 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:52.236424 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:52.248188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:52.249886 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:52.260856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1374/2752] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:07:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:07:56.194991 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69d9958-f62a-43ff-8600-28b0d751364e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:56.195495 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69d9958-f62a-43ff-8600-28b0d751364e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:56.195775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69d9958-f62a-43ff-8600-28b0d751364e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:56.229336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f69d9958-f62a-43ff-8600-28b0d751364e None None] MFA Rules not processed for user `772659a1932d45dea2c5120c4497314b`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:07:56.230775 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f69d9958-f62a-43ff-8600-28b0d751364e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:07:56.345968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1379/2753] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:07:56 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:07:56.349222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:56.367804 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c895bb2-7300-46ea-99c6-ff1958abc976 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:56.373115 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:56.374011 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:56.374288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:56.374557 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:56.375195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:56.377128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:56.408834 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1375/2754] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:07:56 2022] POST /identity/v3/projects => generated 379 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:07:56.412965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69d9958-f62a-43ff-8600-28b0d751364e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:56.426746 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69d9958-f62a-43ff-8600-28b0d751364e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:56.458814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:07:56.501443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:56.502058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:56.502342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:56.502613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:56.503799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:56.506315 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:56.536179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1380/2755] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:07:56 2022] POST /identity/v3/users => generated 296 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:07:56.540205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:56.552922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d7287-151b-4163-b4b1-d819a0e19601 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:56.558832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:56.559428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:56.559705 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:56.559995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:56.560643 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:56.562225 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:56.573717 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1376/2756] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:07:56.577861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:56.588329 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0ac809-0aa5-4936-be45-f95d1dde0fa6 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:56.593157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:56.593808 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:56.594089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:56.594366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] PATH_INFO: `/v3/projects/ac353b06b5524c06b9336411462ae8ad/users/1d0974ede08a4e8fa35ef2e72a63833a/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:56.611662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac353b06b5524c06b9336411462ae8ad, user_id=1d0974ede08a4e8fa35ef2e72a63833a, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:56.614986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:56.630000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1381/2757] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:07:56 2022] PUT /identity/v3/projects/ac353b06b5524c06b9336411462ae8ad/users/1d0974ede08a4e8fa35ef2e72a63833a/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:07:56.633970 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:56.645288 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e98aac-2146-4e59-8e27-5a4c9ea1af17 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:56.707428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00649f03-63e8-4f93-85bb-4919d02cace8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:56.708061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00649f03-63e8-4f93-85bb-4919d02cace8 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:56.708333 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00649f03-63e8-4f93-85bb-4919d02cace8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:56.708599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00649f03-63e8-4f93-85bb-4919d02cace8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:56.709234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00649f03-63e8-4f93-85bb-4919d02cace8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:56.710823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00649f03-63e8-4f93-85bb-4919d02cace8 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:56.722279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1377/2758] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:07:56.725907 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:07:56.737494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccae7e08-110f-4964-aa7e-9a0301bdb10b admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:07:56.750320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:07:56.751145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:07:56.751433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:07:56.751705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] PATH_INFO: `/v3/projects/ac353b06b5524c06b9336411462ae8ad/users/1d0974ede08a4e8fa35ef2e72a63833a/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:07:56.754820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac353b06b5524c06b9336411462ae8ad, user_id=1d0974ede08a4e8fa35ef2e72a63833a, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:07:56.761416 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:07:56.772748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1382/2759] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:07:56 2022] PUT /identity/v3/projects/ac353b06b5524c06b9336411462ae8ad/users/1d0974ede08a4e8fa35ef2e72a63833a/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:08:01.007484 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc215248-547a-44f7-a9e4-c7e996c750b7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:08:01.008104 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc215248-547a-44f7-a9e4-c7e996c750b7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:08:01.008218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc215248-547a-44f7-a9e4-c7e996c750b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:08:01.038177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-bc215248-547a-44f7-a9e4-c7e996c750b7 None None] MFA Rules not processed for user `1d0974ede08a4e8fa35ef2e72a63833a`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:08:01.039068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-bc215248-547a-44f7-a9e4-c7e996c750b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:08:01.111383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1378/2760] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:08:01 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:08:01.131193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:08:01.134941 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:08:01.214403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d463c877-87e8-4680-a28c-c0733b492a64 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:08:01.219873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:08:01.220506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:08:01.220784 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:08:01.221053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:08:01.280251 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:08:01.281833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:08:01.296107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1383/2761] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:08:01 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:08:11.108332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=81,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:08:11.294045 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:08:15.799890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a416b1eb-c7c4-48d0-9a2e-24f9d0f645d4 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:08:15.800928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a416b1eb-c7c4-48d0-9a2e-24f9d0f645d4 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:08:15.801056 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a416b1eb-c7c4-48d0-9a2e-24f9d0f645d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:08:15.820220 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-a416b1eb-c7c4-48d0-9a2e-24f9d0f645d4 None None] MFA Rules not processed for user `86bc1b5163f4453db4469fef90f27344`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:08:15.821415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-a416b1eb-c7c4-48d0-9a2e-24f9d0f645d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:08:15.874947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1379/2762] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:08:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:08:16.175889 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:08:16.177418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:08:16.232749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcc2939-9a4f-4241-bfb1-fc3016d83863 service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:08:16.236503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:08:16.237155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:08:16.237442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:08:16.237717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:08:16.239450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:08:16.249274 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:08:16.250678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:08:16.258964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1384/2763] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:08:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:08:24.157093 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:08:24.157601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:08:24.157702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:08:24.176264 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] MFA Rules not processed for user `772659a1932d45dea2c5120c4497314b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:08:24.177474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:08:24.184200 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1380/2764] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:08:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:08:26.255000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=26 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:08:34.170783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=28 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:08:44.055346 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:08:44.064098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065e7ae0-4c8d-42da-a6c8-5aa909a97df4 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:08:44.068374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:08:44.068960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:08:44.069241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:08:44.069514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:08:44.078297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:08:44.079696 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:08:44.089078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1385/2765] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:08:44 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:08:54.085059 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:09:44.572529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:09:44.574013 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:09:44.590148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5feb913-acda-43e3-b1bb-8544affdc1bf None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:09:44.596336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:09:44.596924 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:09:44.597199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:09:44.597467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:09:44.599121 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:09:44.608950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:09:44.610507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:09:44.619271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1381/2766] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:09:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:09:46.055875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:09:46.057272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:09:46.113443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6147d899-08ef-4c19-a24a-2fab2f4b5a5b service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:09:46.116993 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:09:46.117586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:09:46.117861 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:09:46.118127 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:09:46.119844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:09:46.133373 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:09:46.134773 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:09:46.146653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1386/2767] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:09:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:09:53.971500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:09:53.980629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba58835-ae26-4ea7-991c-59e23a52251b service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:09:53.984194 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6125527e-544e-4e35-829a-d22e3c3dfb7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:09:53.984793 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6125527e-544e-4e35-829a-d22e3c3dfb7e service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:09:53.985049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6125527e-544e-4e35-829a-d22e3c3dfb7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:09:53.985348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6125527e-544e-4e35-829a-d22e3c3dfb7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:09:53.987208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6125527e-544e-4e35-829a-d22e3c3dfb7e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:09:54.000281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6125527e-544e-4e35-829a-d22e3c3dfb7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:09:54.002182 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6125527e-544e-4e35-829a-d22e3c3dfb7e service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:09:54.012858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1382/2768] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:09:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:09:56.140694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=26 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:09:58.741446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:09:58.756481 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1426e1-c746-40c7-a82c-f13265dca701 service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:09:58.762149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:09:58.762762 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:09:58.763036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:09:58.763305 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:09:58.765093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:09:58.779169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:09:58.780788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:09:58.792281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1387/2769] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:09:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:10:02.094357 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dc7c49-6e9b-4c85-8eb4-198ce3ab34af None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:10:02.094839 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dc7c49-6e9b-4c85-8eb4-198ce3ab34af None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:10:02.094948 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dc7c49-6e9b-4c85-8eb4-198ce3ab34af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:10:02.113707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b4dc7c49-6e9b-4c85-8eb4-198ce3ab34af None None] MFA Rules not processed for user `86bc1b5163f4453db4469fef90f27344`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:10:02.114906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b4dc7c49-6e9b-4c85-8eb4-198ce3ab34af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:10:02.121707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1383/2770] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:10:02 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:10:04.510788 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:10:04.521213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e305002-8122-4b92-9bcc-1c2c4d6ca830 service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:10:04.524800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b1b357-2cce-4000-88b7-47990683ef73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:10:04.525521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b1b357-2cce-4000-88b7-47990683ef73 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:10:04.525799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b1b357-2cce-4000-88b7-47990683ef73 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:10:04.526068 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b1b357-2cce-4000-88b7-47990683ef73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:10:04.527802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-71b1b357-2cce-4000-88b7-47990683ef73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:10:04.538283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b1b357-2cce-4000-88b7-47990683ef73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:10:04.539780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b1b357-2cce-4000-88b7-47990683ef73 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:10:04.549875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1388/2771] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:10:04 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:10:09.294308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4dc7c49-6e9b-4c85-8eb4-198ce3ab34af None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:10:09.310962 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4dc7c49-6e9b-4c85-8eb4-198ce3ab34af None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:10:09.319466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:10:09.320111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:10:09.320390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:10:09.320658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:10:09.322429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:10:09.335444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:10:09.337060 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:10:09.348310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1384/2772] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:10:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:10:14.544780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=12 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:10:19.343323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=21 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:05.226518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:05.227061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:05.227061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:05.246756 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] MFA Rules not processed for user `772659a1932d45dea2c5120c4497314b`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:05.248234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:05.255781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1389/2773] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:15.241447 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:44.689788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:44.698960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80372e7-c06b-49e7-828e-fdf2656b527f service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:44.702642 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:44.703226 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:44.703502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:44.703768 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:44.706879 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:44.769160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:44.770583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:44.780575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1385/2774] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:11:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:11:47.538880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:47.541080 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:47.596317 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:47.598392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:47.614205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6516abf7-a38f-4198-8621-982391782a28 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:47.622626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:47.624231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:47.624509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:47.624776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] PATH_INFO: `/v3/users/86bc1b5163f4453db4469fef90f27344` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:47.625995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=86bc1b5163f4453db4469fef90f27344)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:47.627894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:47.635152 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fab199-b026-4e6e-ae4d-61e780a1e661 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:47.642652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:47.642652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:47.642652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:47.642652 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:47.643694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:47.656664 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:47.658541 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:47.671908 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1390/2775] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:11:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:11:47.679509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86bc1b5163f4453db4469fef90f27344'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:47.680462 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86bc1b5163f4453db4469fef90f27344'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:47.687862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86bc1b5163f4453db4469fef90f27344'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:47.688684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86bc1b5163f4453db4469fef90f27344'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:47.704529 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-450c1fba-47b4-4db7-bb49-5b7a31f00c3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86bc1b5163f4453db4469fef90f27344'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:47.717748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1386/2776] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:11:47 2022] DELETE /identity/v3/users/86bc1b5163f4453db4469fef90f27344 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:47.891498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:47.901868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b85856f-1393-4f91-a1be-b5666489962e service placement] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:47.964461 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:47.965093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:47.965371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:47.965639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] PATH_INFO: `/v3/projects/663bde66131d47159ffa09ad33971895` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:47.967388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] RBAC: Authorizing `identity:delete_project(project_id=663bde66131d47159ffa09ad33971895)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:47.969362 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.015580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '663bde66131d47159ffa09ad33971895', 'request_id': 'req-89539776-639d-4683-b52a-c39a4c42f30f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:48.021769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '663bde66131d47159ffa09ad33971895', 'request_id': 'req-89539776-639d-4683-b52a-c39a4c42f30f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:48.027356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '663bde66131d47159ffa09ad33971895', 'request_id': 'req-89539776-639d-4683-b52a-c39a4c42f30f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:48.040406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] The token cache is being invalidate because project 663bde66131d47159ffa09ad33971895 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:11:48.040691 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:48.041319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-89539776-639d-4683-b52a-c39a4c42f30f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:48.042765 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1391/2777] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:11:47 2022] DELETE /identity/v3/projects/663bde66131d47159ffa09ad33971895 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:48.048826 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5241fbd4-8aa1-447b-ac38-6a93ab4b49fe None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.049118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5241fbd4-8aa1-447b-ac38-6a93ab4b49fe None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.049391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5241fbd4-8aa1-447b-ac38-6a93ab4b49fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.068908 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-5241fbd4-8aa1-447b-ac38-6a93ab4b49fe None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:48.070036 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5241fbd4-8aa1-447b-ac38-6a93ab4b49fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:48.129604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1387/2778] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:48.137586 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257f8af-6dd1-4a57-88e5-13f33ef9724f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.137876 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257f8af-6dd1-4a57-88e5-13f33ef9724f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.138149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257f8af-6dd1-4a57-88e5-13f33ef9724f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.159713 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-8257f8af-6dd1-4a57-88e5-13f33ef9724f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:48.160960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-8257f8af-6dd1-4a57-88e5-13f33ef9724f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:48.168986 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1392/2779] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:48.173281 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5241fbd4-8aa1-447b-ac38-6a93ab4b49fe None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.183345 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5241fbd4-8aa1-447b-ac38-6a93ab4b49fe None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.188580 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.189177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.189448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.189718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.190354 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:11:48.190645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.192245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.193944 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:48.207490 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1388/2780] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:11:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:11:48.212391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8257f8af-6dd1-4a57-88e5-13f33ef9724f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.222283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8257f8af-6dd1-4a57-88e5-13f33ef9724f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.226887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.227489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.227766 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.228069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.228720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.230597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.241722 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:48.260534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1393/2781] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:11:48.265129 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.275271 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f5bd9-8a97-4979-8580-45adac660f77 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.330112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.330621 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.330895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.331160 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.331766 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.333684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.354072 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1389/2782] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/users => generated 286 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:11:48.358912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.369061 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf0bf42-8a54-4fb8-9e25-b6915e83b928 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.374638 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.375223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.375500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.375770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.376435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.378008 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.387653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1394/2783] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:48.391566 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.401395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca0a45d-34ad-49de-859f-fd2a2780e5b3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.406512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37296c08-b739-4e18-aeb4-3b151b9e1242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.407087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37296c08-b739-4e18-aeb4-3b151b9e1242 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.407365 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37296c08-b739-4e18-aeb4-3b151b9e1242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.407630 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37296c08-b739-4e18-aeb4-3b151b9e1242 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.408286 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37296c08-b739-4e18-aeb4-3b151b9e1242 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.409878 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37296c08-b739-4e18-aeb4-3b151b9e1242 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.420543 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1390/2784] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:48.424556 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.434316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cb258f-d203-4f43-abbd-9135f94ce76c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.439668 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.440330 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.440611 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.440881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] PATH_INFO: `/v3/projects/5afd5859dfa543ac836c925a18c7986c/users/e661dc691bab481ca37471a62aeadb7f/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.457142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] RBAC: Authorizing `identity:create_grant(project_id=5afd5859dfa543ac836c925a18c7986c, user_id=e661dc691bab481ca37471a62aeadb7f, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.459777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.473275 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1395/2785] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:11:48 2022] PUT /identity/v3/projects/5afd5859dfa543ac836c925a18c7986c/users/e661dc691bab481ca37471a62aeadb7f/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:11:48.484680 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59063762-c2fe-48bc-9c9a-a053623a3a7e None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.484963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59063762-c2fe-48bc-9c9a-a053623a3a7e None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.485230 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59063762-c2fe-48bc-9c9a-a053623a3a7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.512785 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-59063762-c2fe-48bc-9c9a-a053623a3a7e None None] MFA Rules not processed for user `e661dc691bab481ca37471a62aeadb7f`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:48.514896 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-59063762-c2fe-48bc-9c9a-a053623a3a7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:48.578519 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1391/2786] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:48.583091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.645168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271ee6eb-aeb3-4893-8b0d-6a1be5ef5c3f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.649785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.650379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.650665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.650933 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.651565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.653486 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.679954 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1396/2787] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/projects => generated 371 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:11:48.684175 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59063762-c2fe-48bc-9c9a-a053623a3a7e None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.695253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59063762-c2fe-48bc-9c9a-a053623a3a7e None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.752703 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.753294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.753571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.753838 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.754463 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.756423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.765934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:48.778460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1392/2788] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:11:48.782716 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.793225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d8f1b6-ee6a-423e-aa07-f02de535b762 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.798975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.799578 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.800027 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.800306 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.800978 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.802791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.815388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1397/2789] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:48.819631 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.829640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867e178a-413f-4f74-a3b2-bed4a2faf84a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.834439 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.835012 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.835289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.835565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] PATH_INFO: `/v3/projects/d104b040c74345f89ec9ca57efbe6b2e/users/315371e45dd5449aa62bba9458f33111/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.850109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d104b040c74345f89ec9ca57efbe6b2e, user_id=315371e45dd5449aa62bba9458f33111, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.852802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.865969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1393/2790] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:11:48 2022] PUT /identity/v3/projects/d104b040c74345f89ec9ca57efbe6b2e/users/315371e45dd5449aa62bba9458f33111/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:48.869960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.880707 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e61ee0f-7b43-402e-809c-ddbe03004b34 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.936913 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47bc758-1e36-492a-939e-2f45ad1915af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.937503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47bc758-1e36-492a-939e-2f45ad1915af admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.937779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47bc758-1e36-492a-939e-2f45ad1915af admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.938046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47bc758-1e36-492a-939e-2f45ad1915af admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.938672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47bc758-1e36-492a-939e-2f45ad1915af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.940282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47bc758-1e36-492a-939e-2f45ad1915af admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.949824 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1398/2791] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:48 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:48.954431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:48.964657 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea0bdd9-4f7a-463f-ada1-8be54f0c5659 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:48.969469 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:48.970103 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.970379 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.970645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] PATH_INFO: `/v3/projects/d104b040c74345f89ec9ca57efbe6b2e/users/315371e45dd5449aa62bba9458f33111/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:48.972564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d104b040c74345f89ec9ca57efbe6b2e, user_id=315371e45dd5449aa62bba9458f33111, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:48.975166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:48.985718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1394/2792] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:11:48 2022] PUT /identity/v3/projects/d104b040c74345f89ec9ca57efbe6b2e/users/315371e45dd5449aa62bba9458f33111/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:48.995873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f00713-c811-407b-93ce-79ef6d4f02f4 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:48.996186 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f00713-c811-407b-93ce-79ef6d4f02f4 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:48.996454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f00713-c811-407b-93ce-79ef6d4f02f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:49.024518 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-72f00713-c811-407b-93ce-79ef6d4f02f4 None None] MFA Rules not processed for user `315371e45dd5449aa62bba9458f33111`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:49.025700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-72f00713-c811-407b-93ce-79ef6d4f02f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:49.037943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:49.097319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1399/2793] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:48 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:49.109077 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:49.110791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:49.166624 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43eb82c5-b69f-4712-8f8b-5be1d68390b1 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:49.170997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbe5023-9c78-46c0-b0ce-b4eddaa4582f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:49.171583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbe5023-9c78-46c0-b0ce-b4eddaa4582f service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:49.171858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbe5023-9c78-46c0-b0ce-b4eddaa4582f service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:49.172152 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbe5023-9c78-46c0-b0ce-b4eddaa4582f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:49.181073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbe5023-9c78-46c0-b0ce-b4eddaa4582f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:49.182478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcbe5023-9c78-46c0-b0ce-b4eddaa4582f service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:49.192249 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1395/2794] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:11:49 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:11:49.581866 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f00713-c811-407b-93ce-79ef6d4f02f4 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:49.642609 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f00713-c811-407b-93ce-79ef6d4f02f4 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:49.647077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:49.647692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:49.647998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:49.648270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] PATH_INFO: `/v3/users/e661dc691bab481ca37471a62aeadb7f` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:49.649370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e661dc691bab481ca37471a62aeadb7f)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:49.651258 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:49.696573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e661dc691bab481ca37471a62aeadb7f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:49.697246 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e661dc691bab481ca37471a62aeadb7f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:49.697852 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e661dc691bab481ca37471a62aeadb7f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:49.705437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e661dc691bab481ca37471a62aeadb7f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:49.717446 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e661dc691bab481ca37471a62aeadb7f'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:49.729684 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1400/2795] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:11:49 2022] DELETE /identity/v3/users/e661dc691bab481ca37471a62aeadb7f => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:49.734764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:49.735278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:49.735278 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:49.754667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:49.755854 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:49.813421 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1396/2796] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:49.823148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:49.824596 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:49.858600 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:49.883567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4735cfb-05d1-4a19-b402-b1b1aa87498f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:49.887219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:49.887815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:49.888149 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:49.888602 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:49.897647 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:49.899047 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:49.909110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1401/2797] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:11:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:11:50.170585 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:50.172111 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:50.184905 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e29a93-5c23-49a3-9740-260eb8174c46 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:50.190232 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:50.190858 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:50.191135 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:50.191408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] PATH_INFO: `/v3/projects/5afd5859dfa543ac836c925a18c7986c` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:50.192560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5afd5859dfa543ac836c925a18c7986c)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:50.194467 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:50.241047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5afd5859dfa543ac836c925a18c7986c', 'request_id': 'req-4254f968-b357-4b6d-b9bf-eca4b02b5463'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.242946 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=4,keystone:SELECT=92,keystone:DELETE=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:50.249165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5afd5859dfa543ac836c925a18c7986c', 'request_id': 'req-4254f968-b357-4b6d-b9bf-eca4b02b5463'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.255957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5afd5859dfa543ac836c925a18c7986c', 'request_id': 'req-4254f968-b357-4b6d-b9bf-eca4b02b5463'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.268832 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] The token cache is being invalidate because project 5afd5859dfa543ac836c925a18c7986c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:11:50.269116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.269736 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.271205 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1397/2798] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:11:50 2022] DELETE /identity/v3/projects/5afd5859dfa543ac836c925a18c7986c => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:50.275233 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:50.284958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b8beeb-cd55-427b-b982-2528d508e55b service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:50.340949 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:50.342168 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:50.342428 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:50.342700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] PATH_INFO: `/v3/users/315371e45dd5449aa62bba9458f33111` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:50.343814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=315371e45dd5449aa62bba9458f33111)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:50.345761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:50.391084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '315371e45dd5449aa62bba9458f33111'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.391761 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '315371e45dd5449aa62bba9458f33111'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.392358 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '315371e45dd5449aa62bba9458f33111'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.399194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '315371e45dd5449aa62bba9458f33111'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.411741 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b61a788f-5ebe-416e-b231-edb86d529e59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '315371e45dd5449aa62bba9458f33111'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.423739 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1402/2799] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:11:50 2022] DELETE /identity/v3/users/315371e45dd5449aa62bba9458f33111 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:50.696245 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:50.697787 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:50.711166 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4254f968-b357-4b6d-b9bf-eca4b02b5463 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:50.772393 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:50.773108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:50.773280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:50.773558 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] PATH_INFO: `/v3/projects/d104b040c74345f89ec9ca57efbe6b2e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:50.774683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d104b040c74345f89ec9ca57efbe6b2e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:50.776623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:50.823163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd104b040c74345f89ec9ca57efbe6b2e', 'request_id': 'req-260f73c6-c803-4227-a57e-0e0de7c79e22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.829488 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd104b040c74345f89ec9ca57efbe6b2e', 'request_id': 'req-260f73c6-c803-4227-a57e-0e0de7c79e22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.835177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd104b040c74345f89ec9ca57efbe6b2e', 'request_id': 'req-260f73c6-c803-4227-a57e-0e0de7c79e22'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.847759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] The token cache is being invalidate because project d104b040c74345f89ec9ca57efbe6b2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:11:50.848066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.848677 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-260f73c6-c803-4227-a57e-0e0de7c79e22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:11:50.850153 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1398/2800] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:11:50 2022] DELETE /identity/v3/projects/d104b040c74345f89ec9ca57efbe6b2e => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:50.855464 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea0cfd-6a9d-4f34-8124-d85216f42c8e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:50.855980 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea0cfd-6a9d-4f34-8124-d85216f42c8e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:50.856082 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea0cfd-6a9d-4f34-8124-d85216f42c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:50.876469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-c4ea0cfd-6a9d-4f34-8124-d85216f42c8e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:50.877646 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-c4ea0cfd-6a9d-4f34-8124-d85216f42c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:50.924210 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:50.938592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1403/2801] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:50.946254 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85121373-8370-4176-bedd-d33e314a179f None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:50.946476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85121373-8370-4176-bedd-d33e314a179f None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:50.946744 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85121373-8370-4176-bedd-d33e314a179f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:50.965771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-85121373-8370-4176-bedd-d33e314a179f None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:50.967628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-85121373-8370-4176-bedd-d33e314a179f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:50.975761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1399/2802] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:50.980054 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea0cfd-6a9d-4f34-8124-d85216f42c8e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:50.990802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea0cfd-6a9d-4f34-8124-d85216f42c8e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:50.995363 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:50.995958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:50.996254 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:50.996519 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:50.997442 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:11:50.997884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:50.999627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.011785 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1404/2803] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:11:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:11:51.016841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85121373-8370-4176-bedd-d33e314a179f None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.026010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85121373-8370-4176-bedd-d33e314a179f None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.031239 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.031814 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.032120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.032387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.033006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.034902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.062140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1400/2804] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:11:51 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:11:51.066250 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.077307 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1b7a78-b2d2-4c85-b25a-fc455d8a4ae7 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.133386 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.133983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.134263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.134562 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.135185 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.137113 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.158885 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1405/2805] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:11:51 2022] POST /identity/v3/users => generated 288 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:11:51.162965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.173891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4886c26e-8cbd-4173-8fdd-88acffe14498 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.178608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.179229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.179500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.179770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.180436 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.182033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.192433 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1401/2806] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:51.196571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.207153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fc5d9c-8152-46b2-a4c6-9e948374e40a admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.211763 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1192d51d-f812-4fdf-9ca1-09fa3fd13373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.212377 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1192d51d-f812-4fdf-9ca1-09fa3fd13373 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.212655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1192d51d-f812-4fdf-9ca1-09fa3fd13373 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.212922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1192d51d-f812-4fdf-9ca1-09fa3fd13373 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.213553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1192d51d-f812-4fdf-9ca1-09fa3fd13373 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.215128 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1192d51d-f812-4fdf-9ca1-09fa3fd13373 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.227414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1406/2807] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:51.231649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.241644 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96bd1ee-bc80-4861-ae5f-707279426438 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.246340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.247235 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.247600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.247809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] PATH_INFO: `/v3/projects/663e9b4eacde426db202196157a6fe01/users/2299c91751584129a2787f870e89e02a/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.262087 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] RBAC: Authorizing `identity:create_grant(project_id=663e9b4eacde426db202196157a6fe01, user_id=2299c91751584129a2787f870e89e02a, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.265391 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.277922 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1402/2808] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:11:51 2022] PUT /identity/v3/projects/663e9b4eacde426db202196157a6fe01/users/2299c91751584129a2787f870e89e02a/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:51.290212 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df021c-d3d8-481c-bcc4-72805ebe5046 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.290499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df021c-d3d8-481c-bcc4-72805ebe5046 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.290771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0df021c-d3d8-481c-bcc4-72805ebe5046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.318032 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-b0df021c-d3d8-481c-bcc4-72805ebe5046 None None] MFA Rules not processed for user `2299c91751584129a2787f870e89e02a`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:51.319238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b0df021c-d3d8-481c-bcc4-72805ebe5046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:51.382498 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1407/2809] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:51 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:51.387140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.396927 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:51.452229 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f341ab-95ee-4356-8849-de4ff4bfc910 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.456843 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.457431 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.457711 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.457983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.458614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.460544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.486791 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1403/2810] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:11:51 2022] POST /identity/v3/projects => generated 377 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:11:51.491091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df021c-d3d8-481c-bcc4-72805ebe5046 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.503299 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0df021c-d3d8-481c-bcc4-72805ebe5046 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.548049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:51.561350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.562046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.562327 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.562614 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.563213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.565282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.586266 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1408/2811] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:11:51 2022] POST /identity/v3/users => generated 295 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:11:51.590228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.600411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e700bd-6c15-497e-ba62-32311303430e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.604958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.605548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.605823 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.606090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.606723 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.608349 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.618553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1404/2812] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:51.623409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.633213 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befe4eb1-4858-4711-bd07-cf35b059fda3 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.637714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.638943 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.639220 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.639485 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] PATH_INFO: `/v3/projects/981119dae15042a383f3a08aff171b0c/users/716f3b9ffa42406387b866e0e21f89b5/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.653721 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=981119dae15042a383f3a08aff171b0c, user_id=716f3b9ffa42406387b866e0e21f89b5, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.656412 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.668809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1409/2813] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:11:51 2022] PUT /identity/v3/projects/981119dae15042a383f3a08aff171b0c/users/716f3b9ffa42406387b866e0e21f89b5/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:11:51.673026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.682743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df94395-bdb2-48ef-9da2-7956eedbf864 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.737429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eff0c75-ab0d-4455-a20f-a5a61a146882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.737965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff0c75-ab0d-4455-a20f-a5a61a146882 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.738240 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff0c75-ab0d-4455-a20f-a5a61a146882 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.738506 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff0c75-ab0d-4455-a20f-a5a61a146882 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.739134 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff0c75-ab0d-4455-a20f-a5a61a146882 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.740830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff0c75-ab0d-4455-a20f-a5a61a146882 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.751805 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1405/2814] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:11:51 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:11:51.756120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.766895 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a36b34d-b90e-455d-bfc4-ccc3ad0617d9 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.771523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.772318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.772603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.772880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] PATH_INFO: `/v3/projects/981119dae15042a383f3a08aff171b0c/users/716f3b9ffa42406387b866e0e21f89b5/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.774967 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=981119dae15042a383f3a08aff171b0c, user_id=716f3b9ffa42406387b866e0e21f89b5, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:51.777670 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:51.788066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1410/2815] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:11:51 2022] PUT /identity/v3/projects/981119dae15042a383f3a08aff171b0c/users/716f3b9ffa42406387b866e0e21f89b5/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:11:51.799019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b2c96-5c33-4c5a-acd9-e2392e3d8e54 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.799521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b2c96-5c33-4c5a-acd9-e2392e3d8e54 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.799682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73b2c96-5c33-4c5a-acd9-e2392e3d8e54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:51.828504 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e73b2c96-5c33-4c5a-acd9-e2392e3d8e54 None None] MFA Rules not processed for user `716f3b9ffa42406387b866e0e21f89b5`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:11:51.829697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e73b2c96-5c33-4c5a-acd9-e2392e3d8e54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:51.896762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1406/2816] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:11:51 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:11:51.907726 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:51.909469 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:11:51.966732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceb8c88-3e9e-4cc1-8a9e-3a3fc326beb0 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:51.970396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:51.970989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:51.971269 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:51.971538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:52.028865 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:52.030272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:52.039069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1411/2817] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:11:51 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:11:58.728138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b2c96-5c33-4c5a-acd9-e2392e3d8e54 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:11:58.731788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:11:58.749190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73b2c96-5c33-4c5a-acd9-e2392e3d8e54 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:11:58.752867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:11:58.753451 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:11:58.753721 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:11:58.753981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:11:58.784341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:11:58.785776 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:11:58.795496 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1407/2818] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:11:58 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:02.034985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=70,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:08.790866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=19 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:17.721969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:17.723350 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:17.862612 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34882be9-3ab8-4666-b536-bdd2a99eff2d service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:17.867719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a3acaf-6f7a-4ec2-9dbf-5e519e17d46b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:17.868435 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3acaf-6f7a-4ec2-9dbf-5e519e17d46b service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:17.868733 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3acaf-6f7a-4ec2-9dbf-5e519e17d46b service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:17.868887 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a3acaf-6f7a-4ec2-9dbf-5e519e17d46b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:17.870590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-89a3acaf-6f7a-4ec2-9dbf-5e519e17d46b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:17.882279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3acaf-6f7a-4ec2-9dbf-5e519e17d46b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:17.883833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a3acaf-6f7a-4ec2-9dbf-5e519e17d46b service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:17.894496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1412/2819] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:26.928864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:27.009601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64568784-7aaa-4014-94bb-0e722b41e0d8 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:27.014218 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:27.014857 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:27.015132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:27.015401 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] PATH_INFO: `/v3/users/2299c91751584129a2787f870e89e02a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:27.016544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2299c91751584129a2787f870e89e02a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:27.019052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:27.063819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2299c91751584129a2787f870e89e02a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.064949 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2299c91751584129a2787f870e89e02a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.071628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2299c91751584129a2787f870e89e02a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.072250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2299c91751584129a2787f870e89e02a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.084256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2299c91751584129a2787f870e89e02a'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.096982 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1408/2820] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:12:26 2022] DELETE /identity/v3/users/2299c91751584129a2787f870e89e02a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:27.101592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:27.102094 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:27.102205 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:27.142243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:27.144142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:27.201462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1413/2821] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:12:27.211774 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:27.213188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:27.269212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecc426b-06f1-43c6-b0bd-39c0280e264e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:27.272856 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:27.273444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:27.273720 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:27.273981 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:27.282841 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:27.284448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:27.293700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1409/2822] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:27.554603 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:27.556034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:27.568853 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205a8c22-91d9-418d-a7da-0ef5c91801df None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:27.573457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:27.574088 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:27.574371 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:27.574643 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] PATH_INFO: `/v3/projects/663e9b4eacde426db202196157a6fe01` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:27.575776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] RBAC: Authorizing `identity:delete_project(project_id=663e9b4eacde426db202196157a6fe01)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:27.577694 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:27.584479 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:27.625825 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '663e9b4eacde426db202196157a6fe01', 'request_id': 'req-e224deb4-098e-4ddd-8998-bf2b69e4f229'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.632966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '663e9b4eacde426db202196157a6fe01', 'request_id': 'req-e224deb4-098e-4ddd-8998-bf2b69e4f229'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.638444 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '663e9b4eacde426db202196157a6fe01', 'request_id': 'req-e224deb4-098e-4ddd-8998-bf2b69e4f229'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.650434 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] The token cache is being invalidate because project 663e9b4eacde426db202196157a6fe01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:12:27.650719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.651320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.652820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1414/2823] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:12:27 2022] DELETE /identity/v3/projects/663e9b4eacde426db202196157a6fe01 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:27.656872 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:27.663607 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:27.667578 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f2f104-d4b9-43ca-949c-f97a282d576b service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:27.726875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:27.727511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:27.727786 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:27.728090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] PATH_INFO: `/v3/users/716f3b9ffa42406387b866e0e21f89b5` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:27.729193 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=716f3b9ffa42406387b866e0e21f89b5)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:27.731080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:27.776370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '716f3b9ffa42406387b866e0e21f89b5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.777000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '716f3b9ffa42406387b866e0e21f89b5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.784493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '716f3b9ffa42406387b866e0e21f89b5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.785139 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '716f3b9ffa42406387b866e0e21f89b5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.797098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-5d3223e0-4276-4f7c-a7e3-8adc4daf7dc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '716f3b9ffa42406387b866e0e21f89b5'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:27.811250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1410/2824] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:12:27 2022] DELETE /identity/v3/users/716f3b9ffa42406387b866e0e21f89b5 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:28.079579 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.081172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:28.095153 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e224deb4-098e-4ddd-8998-bf2b69e4f229 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.153118 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.153754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.154087 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.154277 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] PATH_INFO: `/v3/projects/981119dae15042a383f3a08aff171b0c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.155384 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] RBAC: Authorizing `identity:delete_project(project_id=981119dae15042a383f3a08aff171b0c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.157302 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.205166 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '981119dae15042a383f3a08aff171b0c', 'request_id': 'req-9257ba21-54bd-46ee-96e2-18c9af862ffc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:28.211403 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '981119dae15042a383f3a08aff171b0c', 'request_id': 'req-9257ba21-54bd-46ee-96e2-18c9af862ffc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:28.216966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '981119dae15042a383f3a08aff171b0c', 'request_id': 'req-9257ba21-54bd-46ee-96e2-18c9af862ffc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:28.228956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] The token cache is being invalidate because project 981119dae15042a383f3a08aff171b0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:12:28.229245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:28.229869 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-9257ba21-54bd-46ee-96e2-18c9af862ffc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:28.231345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1415/2825] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:12:28 2022] DELETE /identity/v3/projects/981119dae15042a383f3a08aff171b0c => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:28.236861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfbd82f-207b-4016-a3fc-4c4c30c0e00a None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.237447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfbd82f-207b-4016-a3fc-4c4c30c0e00a None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.237447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfbd82f-207b-4016-a3fc-4c4c30c0e00a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.285541 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1dfbd82f-207b-4016-a3fc-4c4c30c0e00a None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:28.286726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1dfbd82f-207b-4016-a3fc-4c4c30c0e00a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:28.336952 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:28.346143 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1411/2826] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:28.406719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a243855b-3a0b-42cb-b214-51fd29a40421 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.406983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a243855b-3a0b-42cb-b214-51fd29a40421 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.407245 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a243855b-3a0b-42cb-b214-51fd29a40421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.428270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-a243855b-3a0b-42cb-b214-51fd29a40421 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:28.429499 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a243855b-3a0b-42cb-b214-51fd29a40421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:28.437388 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1416/2827] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:28.441866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfbd82f-207b-4016-a3fc-4c4c30c0e00a None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.451459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfbd82f-207b-4016-a3fc-4c4c30c0e00a None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.456243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.456770 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.457046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.457312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.457961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:12:28.458258 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.460132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.472318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1412/2828] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:12:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:12:28.476535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a243855b-3a0b-42cb-b214-51fd29a40421 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.485622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a243855b-3a0b-42cb-b214-51fd29a40421 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.490973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.491553 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.491832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.492157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.492779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.494653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.510079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:28.524325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1417/2829] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:12:28 2022] POST /identity/v3/projects => generated 401 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:28.528592 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.539110 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfb7f97-7036-4a97-9d20-f453accc5d56 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.594719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.595837 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.596163 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.596423 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.597098 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.599024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.620921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1413/2830] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:12:28 2022] POST /identity/v3/users => generated 301 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:28.625527 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.635968 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4141f122-409b-4c28-984a-f79e1f25b29d admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.640534 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.641124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.641407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.641688 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.642320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.643901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.653665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1418/2831] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:28.657628 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.668150 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b4ea65-2bc1-4a8b-a5a3-473f18da7475 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.672722 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a39378f-a7de-4a36-a5b8-9efb4b756be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.673300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39378f-a7de-4a36-a5b8-9efb4b756be6 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.674140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39378f-a7de-4a36-a5b8-9efb4b756be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.674411 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a39378f-a7de-4a36-a5b8-9efb4b756be6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.675042 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39378f-a7de-4a36-a5b8-9efb4b756be6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.676663 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a39378f-a7de-4a36-a5b8-9efb4b756be6 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.686968 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1414/2832] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:28.691309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.701281 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7914723d-4cc4-4691-982c-fc17e38b19c2 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.706179 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.706901 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.707136 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.707409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] PATH_INFO: `/v3/projects/01f69c351a434283b7067da1a0df1730/users/cea0704a7d654c65a4b5214f55631c21/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.722311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=01f69c351a434283b7067da1a0df1730, user_id=cea0704a7d654c65a4b5214f55631c21, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.724970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.738000 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1419/2833] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:28 2022] PUT /identity/v3/projects/01f69c351a434283b7067da1a0df1730/users/cea0704a7d654c65a4b5214f55631c21/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:28.751964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f86cb-bb69-4b95-985d-d9aab7c02e19 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.752253 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f86cb-bb69-4b95-985d-d9aab7c02e19 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.752522 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350f86cb-bb69-4b95-985d-d9aab7c02e19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.779743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-350f86cb-bb69-4b95-985d-d9aab7c02e19 None None] MFA Rules not processed for user `cea0704a7d654c65a4b5214f55631c21`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:28.780963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-350f86cb-bb69-4b95-985d-d9aab7c02e19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:28.848860 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1415/2834] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:28 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:12:28.853297 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.914348 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be3cda-8675-41d5-8c63-bebc23f9d4ce admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:28.918938 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:28.919528 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:28.919818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:28.920116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:28.920747 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:28.922633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:28.949133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1420/2835] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:12:28 2022] POST /identity/v3/projects => generated 401 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:12:28.952471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f86cb-bb69-4b95-985d-d9aab7c02e19 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:28.963867 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350f86cb-bb69-4b95-985d-d9aab7c02e19 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:29.012512 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:29.020845 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:29.021465 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.021764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.022030 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.022661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:29.024599 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:29.048132 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1416/2836] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:12:28 2022] POST /identity/v3/users => generated 307 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:29.049455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:29.060272 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4170358-394c-4179-978e-9e0087658a06 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:29.064937 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:29.065523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.065802 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.066073 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.066706 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:29.068344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:29.078678 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1421/2837] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:29.083427 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:29.093757 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86ef2e5-f6eb-4c2d-a888-18f3cebcb4d2 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:29.099071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:29.099725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.100035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.100306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] PATH_INFO: `/v3/projects/0d546d50da6946bda8df5df4114f473c/users/9fbc4775f1f5491c9cca244d1d258f5f/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.114622 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d546d50da6946bda8df5df4114f473c, user_id=9fbc4775f1f5491c9cca244d1d258f5f, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:29.117266 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:29.129853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1417/2838] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:29 2022] PUT /identity/v3/projects/0d546d50da6946bda8df5df4114f473c/users/9fbc4775f1f5491c9cca244d1d258f5f/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:29.133723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:29.144285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-842c3918-03c1-481b-accf-3134e33bca2f admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:29.202126 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e64b2b4-d9a7-475d-98bd-1a2a3525a633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:29.202814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e64b2b4-d9a7-475d-98bd-1a2a3525a633 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.202990 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e64b2b4-d9a7-475d-98bd-1a2a3525a633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.203261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e64b2b4-d9a7-475d-98bd-1a2a3525a633 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.203894 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e64b2b4-d9a7-475d-98bd-1a2a3525a633 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:29.206078 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e64b2b4-d9a7-475d-98bd-1a2a3525a633 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:29.215742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1422/2839] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:29.219875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:29.230019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2bb535-3442-452b-b6e8-856b469abf6e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:29.234809 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:29.235441 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.235715 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.236034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] PATH_INFO: `/v3/projects/0d546d50da6946bda8df5df4114f473c/users/9fbc4775f1f5491c9cca244d1d258f5f/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.238018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d546d50da6946bda8df5df4114f473c, user_id=9fbc4775f1f5491c9cca244d1d258f5f, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:29.240661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:29.250895 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1418/2840] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:29 2022] PUT /identity/v3/projects/0d546d50da6946bda8df5df4114f473c/users/9fbc4775f1f5491c9cca244d1d258f5f/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:29.262284 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290669f4-67f4-4883-b49e-9f2905df8dab None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.262650 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290669f4-67f4-4883-b49e-9f2905df8dab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.262931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290669f4-67f4-4883-b49e-9f2905df8dab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.290754 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-290669f4-67f4-4883-b49e-9f2905df8dab None None] MFA Rules not processed for user `9fbc4775f1f5491c9cca244d1d258f5f`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:29.291998 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-290669f4-67f4-4883-b49e-9f2905df8dab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:29.310391 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:29.365115 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1423/2841] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:29.375964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:29.378228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:29.436272 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4885517a-1415-43be-a1b2-1875c09fefcb admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:29.439960 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:29.441362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.441636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.441902 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.450704 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:29.452144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:29.462118 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1419/2842] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:29 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:29.538090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290669f4-67f4-4883-b49e-9f2905df8dab None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:29.547680 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290669f4-67f4-4883-b49e-9f2905df8dab None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:29.551796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b4c84-ae6a-4d0e-9679-bb65259696e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:29.552419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b4c84-ae6a-4d0e-9679-bb65259696e6 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:29.552699 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b4c84-ae6a-4d0e-9679-bb65259696e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:29.552964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b4c84-ae6a-4d0e-9679-bb65259696e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:29.610847 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b4c84-ae6a-4d0e-9679-bb65259696e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:29.612344 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b4c84-ae6a-4d0e-9679-bb65259696e6 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:29.621171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1424/2843] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:29 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:33.683048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:33.744807 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1702f3-2768-44e6-8865-36b3beb98038 service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:33.749446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:33.750053 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:33.750329 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:33.750589 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] PATH_INFO: `/v3/users/cea0704a7d654c65a4b5214f55631c21` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:33.751679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=cea0704a7d654c65a4b5214f55631c21)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:33.753935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:33.797966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cea0704a7d654c65a4b5214f55631c21'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:33.798555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cea0704a7d654c65a4b5214f55631c21'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:33.803820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:33.810915 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cea0704a7d654c65a4b5214f55631c21'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:33.811853 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cea0704a7d654c65a4b5214f55631c21'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:33.823789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cea0704a7d654c65a4b5214f55631c21'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:33.836476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1420/2844] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:12:33 2022] DELETE /identity/v3/users/cea0704a7d654c65a4b5214f55631c21 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:33.840964 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:33.841456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:33.841572 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:33.860786 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:33.861983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:33.920744 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1425/2845] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:33.930935 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:33.932374 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:33.987890 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e5126f-1dd0-437d-b775-62300b4abf6b admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:33.991491 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:33.992109 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:33.992386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:33.992652 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:34.001602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:34.003215 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:34.012455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1421/2846] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:34.279779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:34.281452 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:34.295319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414f8b27-b8d6-4acf-9953-c0254d6f97cc None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:34.300862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:34.301548 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:34.301827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:34.302100 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] PATH_INFO: `/v3/projects/01f69c351a434283b7067da1a0df1730` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:34.303248 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01f69c351a434283b7067da1a0df1730)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:34.305206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:34.351717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01f69c351a434283b7067da1a0df1730', 'request_id': 'req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.352856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:34.360267 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01f69c351a434283b7067da1a0df1730', 'request_id': 'req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.366410 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01f69c351a434283b7067da1a0df1730', 'request_id': 'req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.379196 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] The token cache is being invalidate because project 01f69c351a434283b7067da1a0df1730 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:12:34.379482 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.380130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.381599 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1426/2847] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:12:34 2022] DELETE /identity/v3/projects/01f69c351a434283b7067da1a0df1730 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:34.385707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:34.395151 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe4bd67-22dd-4504-83ab-09fb6b9d05a7 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:34.450934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:34.451544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:34.451819 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:34.452125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] PATH_INFO: `/v3/users/9fbc4775f1f5491c9cca244d1d258f5f` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:34.453250 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fbc4775f1f5491c9cca244d1d258f5f)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:34.455156 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:34.500603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fbc4775f1f5491c9cca244d1d258f5f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.501243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fbc4775f1f5491c9cca244d1d258f5f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.508147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fbc4775f1f5491c9cca244d1d258f5f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.508771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fbc4775f1f5491c9cca244d1d258f5f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.520891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-bf5a6629-c7ca-4b07-9ea9-6832de993c84 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fbc4775f1f5491c9cca244d1d258f5f'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.522508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:34.536756 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1422/2848] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:12:34 2022] DELETE /identity/v3/users/9fbc4775f1f5491c9cca244d1d258f5f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:34.808742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:34.810336 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:34.824133 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9a7aeb-f0a6-4fb8-bba9-ebe1a1a2f968 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:34.883335 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:34.884053 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:34.884311 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:34.884533 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] PATH_INFO: `/v3/projects/0d546d50da6946bda8df5df4114f473c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:34.885622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d546d50da6946bda8df5df4114f473c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:34.887483 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:34.934836 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d546d50da6946bda8df5df4114f473c', 'request_id': 'req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.941090 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d546d50da6946bda8df5df4114f473c', 'request_id': 'req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.946517 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d546d50da6946bda8df5df4114f473c', 'request_id': 'req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.963069 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] The token cache is being invalidate because project 0d546d50da6946bda8df5df4114f473c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:12:34.963356 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.964051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-67d8ba0b-4a22-4ac5-9495-579b1b0ffe3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:34.965531 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1427/2849] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:12:34 2022] DELETE /identity/v3/projects/0d546d50da6946bda8df5df4114f473c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:34.970603 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763711cb-644e-4a88-bd39-f4b8c7ddc74d None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:34.971070 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763711cb-644e-4a88-bd39-f4b8c7ddc74d None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:34.971190 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763711cb-644e-4a88-bd39-f4b8c7ddc74d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:34.991147 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-763711cb-644e-4a88-bd39-f4b8c7ddc74d None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:34.992380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-763711cb-644e-4a88-bd39-f4b8c7ddc74d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:35.050913 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1423/2850] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:35.058666 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15d1112-6299-447f-9568-2a0abfb4f9ab None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.058976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15d1112-6299-447f-9568-2a0abfb4f9ab None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.059264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15d1112-6299-447f-9568-2a0abfb4f9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.078500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-f15d1112-6299-447f-9568-2a0abfb4f9ab None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:35.079679 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f15d1112-6299-447f-9568-2a0abfb4f9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:35.088140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1428/2851] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:35.092375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763711cb-644e-4a88-bd39-f4b8c7ddc74d None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.102847 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763711cb-644e-4a88-bd39-f4b8c7ddc74d None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.107310 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.107917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.108221 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.108492 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.109145 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:12:35.109444 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.111033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.123006 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1424/2852] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:12:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:12:35.127433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15d1112-6299-447f-9568-2a0abfb4f9ab None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.137103 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15d1112-6299-447f-9568-2a0abfb4f9ab None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.141558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.142169 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.142426 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.142690 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.143309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.145880 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.178011 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1429/2853] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:35.182067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.192461 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c017cd-f983-4c51-bc3c-e7cf7fe7261a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.249714 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.250316 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.250595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.250866 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.251500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.254112 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.275626 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1425/2854] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/users => generated 287 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:35.279686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.290084 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e528c67f-6c25-4912-aaa7-2cab44699105 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.294817 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.295406 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.295683 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.296007 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.296662 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.298630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.308828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1430/2855] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:35.312718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.322943 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93d8632-7faf-4d16-a685-425ffc87cb58 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.327508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d4f435-5800-4f08-854a-8046c36592fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.328195 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d4f435-5800-4f08-854a-8046c36592fb admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.328475 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d4f435-5800-4f08-854a-8046c36592fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.328743 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d4f435-5800-4f08-854a-8046c36592fb admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.329370 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d4f435-5800-4f08-854a-8046c36592fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.331734 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d4f435-5800-4f08-854a-8046c36592fb admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.341993 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1426/2856] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:35.346058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.355985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873b70cc-9693-4520-963e-281608f4e3b8 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.360728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.362409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.362770 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.363049 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] PATH_INFO: `/v3/projects/2e479d5262b74882bcebb25d0a46e44a/users/9db9535c9d9f424bb86ff5f3d2157dcc/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.378400 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e479d5262b74882bcebb25d0a46e44a, user_id=9db9535c9d9f424bb86ff5f3d2157dcc, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.381072 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.394450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1431/2857] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:35 2022] PUT /identity/v3/projects/2e479d5262b74882bcebb25d0a46e44a/users/9db9535c9d9f424bb86ff5f3d2157dcc/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:35.405049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3bd29e-6d45-430d-b3df-3953419540d6 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.405343 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3bd29e-6d45-430d-b3df-3953419540d6 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.405617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3bd29e-6d45-430d-b3df-3953419540d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.432687 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-4d3bd29e-6d45-430d-b3df-3953419540d6 None None] MFA Rules not processed for user `9db9535c9d9f424bb86ff5f3d2157dcc`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:35.433887 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-4d3bd29e-6d45-430d-b3df-3953419540d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:35.479210 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:35.500555 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1427/2858] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:35.505676 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.511242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:35.569717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d8a160-d484-4418-8267-6e81c1aabd09 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.574613 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.575206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.575514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.575782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.576465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.578409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.604036 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1432/2859] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/projects => generated 375 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:35.608020 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3bd29e-6d45-430d-b3df-3953419540d6 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.619209 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3bd29e-6d45-430d-b3df-3953419540d6 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.674196 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.675292 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.675650 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.675916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.676563 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.678459 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.698729 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1428/2860] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:35.702834 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.713117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808bddd6-b7a1-4fd2-88cd-a6f89bd79af2 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.717650 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.718231 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.718509 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.718778 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.719405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.721006 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.731453 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1433/2861] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:35.735282 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.745992 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf613e91-8f24-4ad0-bc37-d505518fa402 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.750938 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3922b64-4114-4e12-94dc-af343beef160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.751515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3922b64-4114-4e12-94dc-af343beef160 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.751817 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3922b64-4114-4e12-94dc-af343beef160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.752114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3922b64-4114-4e12-94dc-af343beef160 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.753380 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3922b64-4114-4e12-94dc-af343beef160 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.754939 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3922b64-4114-4e12-94dc-af343beef160 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.764616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1429/2862] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:12:35.768753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.778565 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dccb58-647d-4beb-a52f-6cad4aa8076c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.783206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.783841 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.784257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.784535 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] PATH_INFO: `/v3/projects/58acf7d8a1f54462bd7ec734d62ad7b1/users/43fef0154abd41288e72e38b0d0f5a0c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.798973 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58acf7d8a1f54462bd7ec734d62ad7b1, user_id=43fef0154abd41288e72e38b0d0f5a0c, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:35.802244 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:35.817987 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1434/2863] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:35 2022] PUT /identity/v3/projects/58acf7d8a1f54462bd7ec734d62ad7b1/users/43fef0154abd41288e72e38b0d0f5a0c/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:35.829386 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f908a4f-f9b5-4403-95b3-a40d332be905 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.829684 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f908a4f-f9b5-4403-95b3-a40d332be905 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.829950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f908a4f-f9b5-4403-95b3-a40d332be905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.858781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-8f908a4f-f9b5-4403-95b3-a40d332be905 None None] MFA Rules not processed for user `43fef0154abd41288e72e38b0d0f5a0c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:35.859965 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-8f908a4f-f9b5-4403-95b3-a40d332be905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:35.924402 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1430/2864] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:12:35.929868 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:35.991902 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5351f5d6-9798-4316-b8a3-c404233c1e10 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:35.996731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:35.997319 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:35.997592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:35.997860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:35.998478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:36.000470 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:36.026034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1435/2865] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:12:35 2022] POST /identity/v3/projects => generated 375 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:36.030144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f908a4f-f9b5-4403-95b3-a40d332be905 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:36.041164 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f908a4f-f9b5-4403-95b3-a40d332be905 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:36.091199 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:36.099188 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:36.100016 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:36.100486 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:36.100759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:36.101375 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:36.103279 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:36.123885 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1431/2866] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:12:36 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:36.128398 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:36.138478 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2725e6a-07bd-4afe-a934-8ca178d6c531 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:36.143029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:36.144295 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:36.144570 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:36.144837 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:36.145475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:36.147050 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:36.157326 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1436/2867] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:36.161452 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:36.171471 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e394d29c-d406-464e-b4a1-02b2edd06d5d admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:36.176553 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:36.177131 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:36.177408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:36.177676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] PATH_INFO: `/v3/projects/9acef62abb4147fbb8f6ab8e6e68a6b2/users/166f0d0e2c9f45ec84b423852a5bf3d7/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:36.192799 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] RBAC: Authorizing `identity:create_grant(project_id=9acef62abb4147fbb8f6ab8e6e68a6b2, user_id=166f0d0e2c9f45ec84b423852a5bf3d7, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:36.195428 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:36.208803 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1432/2868] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:36 2022] PUT /identity/v3/projects/9acef62abb4147fbb8f6ab8e6e68a6b2/users/166f0d0e2c9f45ec84b423852a5bf3d7/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:36.212815 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:36.222960 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dd03e9-3fa9-4409-a169-73bd50bcee78 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:36.248225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:36.280051 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d19ae41-c9f5-473a-a704-870511829238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:36.280648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d19ae41-c9f5-473a-a704-870511829238 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:36.280929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d19ae41-c9f5-473a-a704-870511829238 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:36.281202 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d19ae41-c9f5-473a-a704-870511829238 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:36.281833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d19ae41-c9f5-473a-a704-870511829238 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:36.283404 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d19ae41-c9f5-473a-a704-870511829238 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:36.292899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1437/2869] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:36 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:36.297088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:36.307906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9cde1-204a-49bf-83aa-4994457a300f admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:36.312990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:36.313629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:36.313906 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:36.314177 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] PATH_INFO: `/v3/projects/9acef62abb4147fbb8f6ab8e6e68a6b2/users/166f0d0e2c9f45ec84b423852a5bf3d7/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:36.316963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9acef62abb4147fbb8f6ab8e6e68a6b2, user_id=166f0d0e2c9f45ec84b423852a5bf3d7, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:36.319575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:36.329629 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1433/2870] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:36 2022] PUT /identity/v3/projects/9acef62abb4147fbb8f6ab8e6e68a6b2/users/166f0d0e2c9f45ec84b423852a5bf3d7/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:36.339776 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:36.340108 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:36.340389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:36.367992 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] MFA Rules not processed for user `166f0d0e2c9f45ec84b423852a5bf3d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:36.369206 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:36.436471 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1438/2871] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:36 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:36.448138 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:36.449840 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:36.506517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3047783d-d254-4bdb-a0af-f226805e7fe3 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:36.510362 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:36.510983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:36.511256 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:36.511521 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:36.566928 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:36.568781 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:36.577972 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1434/2872] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:38.081976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:38.083454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:38.095193 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea5b9f2-a600-49a0-b2eb-2411f10f184e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:38.099544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:38.100161 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:38.100440 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:38.100710 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:38.155818 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:38.157262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:38.166034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1439/2873] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:38 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:38.700456 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:38.709071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbc5902-176d-4418-b33e-7f973fe66dde service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:38.712696 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a383e1-8192-4739-bcdd-5dde4cee961a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:38.713283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a383e1-8192-4739-bcdd-5dde4cee961a service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:38.713561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a383e1-8192-4739-bcdd-5dde4cee961a service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:38.713830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a383e1-8192-4739-bcdd-5dde4cee961a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:38.722963 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a383e1-8192-4739-bcdd-5dde4cee961a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:38.724396 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a383e1-8192-4739-bcdd-5dde4cee961a service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:38.733482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1435/2874] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:38 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:42.614159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:42.662648 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=100 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:42.676926 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211cae1d-c7b0-4a49-b1ed-8d955f1ab5ba service cinder] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:42.681582 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:42.682264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:42.682545 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:42.682813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] PATH_INFO: `/v3/users/9db9535c9d9f424bb86ff5f3d2157dcc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:42.683916 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9db9535c9d9f424bb86ff5f3d2157dcc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:42.685831 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:42.732521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9db9535c9d9f424bb86ff5f3d2157dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:42.733157 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9db9535c9d9f424bb86ff5f3d2157dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:42.733748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9db9535c9d9f424bb86ff5f3d2157dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:42.740523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9db9535c9d9f424bb86ff5f3d2157dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:42.752308 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9db9535c9d9f424bb86ff5f3d2157dcc'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:42.764883 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1440/2875] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:12:42 2022] DELETE /identity/v3/users/9db9535c9d9f424bb86ff5f3d2157dcc => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:42.770061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:42.770572 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:42.770682 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:42.789268 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:42.790472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:42.835368 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:42.851673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1436/2876] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:42.861389 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:42.863066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:42.920294 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7920032-9adf-4d91-8779-62f5c49e5ce0 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:42.923856 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:42.924475 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:42.924873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:42.925240 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:42.934793 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:42.936241 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:42.946204 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1441/2877] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:43.213472 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:43.215410 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:43.228912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0833121c-3dcc-48e6-b485-3231eed14885 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:43.233438 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:43.234142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:43.234503 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:43.234957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] PATH_INFO: `/v3/projects/2e479d5262b74882bcebb25d0a46e44a` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:43.236623 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e479d5262b74882bcebb25d0a46e44a)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:43.239584 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:43.286377 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e479d5262b74882bcebb25d0a46e44a', 'request_id': 'req-51caedfc-12d5-4a00-b81c-0b953692a718'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.292564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e479d5262b74882bcebb25d0a46e44a', 'request_id': 'req-51caedfc-12d5-4a00-b81c-0b953692a718'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.298027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e479d5262b74882bcebb25d0a46e44a', 'request_id': 'req-51caedfc-12d5-4a00-b81c-0b953692a718'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.310658 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] The token cache is being invalidate because project 2e479d5262b74882bcebb25d0a46e44a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:12:43.310942 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.311700 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.313035 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1437/2878] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:12:43 2022] DELETE /identity/v3/projects/2e479d5262b74882bcebb25d0a46e44a => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:43.317715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:43.327092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca267816-88d7-4755-8f8a-c60a75fffa56 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:43.384333 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:43.384953 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:43.385228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:43.385494 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] PATH_INFO: `/v3/users/43fef0154abd41288e72e38b0d0f5a0c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:43.386591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43fef0154abd41288e72e38b0d0f5a0c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:43.388515 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:43.401433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:43.435604 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43fef0154abd41288e72e38b0d0f5a0c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.436260 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43fef0154abd41288e72e38b0d0f5a0c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.436867 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43fef0154abd41288e72e38b0d0f5a0c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.443810 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43fef0154abd41288e72e38b0d0f5a0c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.455261 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43fef0154abd41288e72e38b0d0f5a0c'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.469171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1442/2879] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:12:43 2022] DELETE /identity/v3/users/43fef0154abd41288e72e38b0d0f5a0c => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:43.739576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:43.741021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:43.756969 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51caedfc-12d5-4a00-b81c-0b953692a718 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:43.817174 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:43.817802 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:43.818076 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:43.818342 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] PATH_INFO: `/v3/projects/58acf7d8a1f54462bd7ec734d62ad7b1` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:43.819507 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58acf7d8a1f54462bd7ec734d62ad7b1)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:43.821413 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:43.868762 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58acf7d8a1f54462bd7ec734d62ad7b1', 'request_id': 'req-d8005c77-6021-4bf7-81bf-0a347bf9b050'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.876325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58acf7d8a1f54462bd7ec734d62ad7b1', 'request_id': 'req-d8005c77-6021-4bf7-81bf-0a347bf9b050'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.883014 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58acf7d8a1f54462bd7ec734d62ad7b1', 'request_id': 'req-d8005c77-6021-4bf7-81bf-0a347bf9b050'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.895493 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] The token cache is being invalidate because project 58acf7d8a1f54462bd7ec734d62ad7b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:12:43.895825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.896523 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:43.898046 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1438/2880] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:12:43 2022] DELETE /identity/v3/projects/58acf7d8a1f54462bd7ec734d62ad7b1 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:43.902285 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:43.903700 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:43.917116 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa29b7b-f0ff-4270-8af3-9ede79c53308 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:43.988431 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:43.989071 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:43.989351 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:43.989625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] PATH_INFO: `/v3/users/166f0d0e2c9f45ec84b423852a5bf3d7` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:43.990777 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=166f0d0e2c9f45ec84b423852a5bf3d7)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:43.993669 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:44.047712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '166f0d0e2c9f45ec84b423852a5bf3d7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.048422 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '166f0d0e2c9f45ec84b423852a5bf3d7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.049015 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '166f0d0e2c9f45ec84b423852a5bf3d7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.057544 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '166f0d0e2c9f45ec84b423852a5bf3d7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.071590 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-e3433b8d-68f0-41f5-ad8c-11e1c3a294ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '166f0d0e2c9f45ec84b423852a5bf3d7'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.085580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1443/2881] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:12:43 2022] DELETE /identity/v3/users/166f0d0e2c9f45ec84b423852a5bf3d7 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:44.578581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:44.580119 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:44.587917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:44.594709 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8005c77-6021-4bf7-81bf-0a347bf9b050 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:44.654383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:44.655003 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:44.655280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:44.655546 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] PATH_INFO: `/v3/projects/9acef62abb4147fbb8f6ab8e6e68a6b2` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:44.656675 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9acef62abb4147fbb8f6ab8e6e68a6b2)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:44.658568 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:44.704075 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9acef62abb4147fbb8f6ab8e6e68a6b2', 'request_id': 'req-b21c4df5-dc99-4c84-8ed3-e2724596fd12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.710608 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9acef62abb4147fbb8f6ab8e6e68a6b2', 'request_id': 'req-b21c4df5-dc99-4c84-8ed3-e2724596fd12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.715894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9acef62abb4147fbb8f6ab8e6e68a6b2', 'request_id': 'req-b21c4df5-dc99-4c84-8ed3-e2724596fd12'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.728679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] The token cache is being invalidate because project 9acef62abb4147fbb8f6ab8e6e68a6b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:12:44.728964 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.729577 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-b21c4df5-dc99-4c84-8ed3-e2724596fd12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:12:44.731043 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1439/2882] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:12:44 2022] DELETE /identity/v3/projects/9acef62abb4147fbb8f6ab8e6e68a6b2 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:44.736379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992350e3-0bf9-4b8f-9981-5ca720627ebe None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:44.736705 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992350e3-0bf9-4b8f-9981-5ca720627ebe None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:44.737655 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992350e3-0bf9-4b8f-9981-5ca720627ebe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:44.764513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-992350e3-0bf9-4b8f-9981-5ca720627ebe None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:44.765708 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-992350e3-0bf9-4b8f-9981-5ca720627ebe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:44.770355 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:44.827771 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1444/2883] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:44.837184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926201a9-f50f-456b-aaf3-48d7ecb38d90 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:44.837681 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926201a9-f50f-456b-aaf3-48d7ecb38d90 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:44.837806 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926201a9-f50f-456b-aaf3-48d7ecb38d90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:44.857034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-926201a9-f50f-456b-aaf3-48d7ecb38d90 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:44.858207 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-926201a9-f50f-456b-aaf3-48d7ecb38d90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:44.868300 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1440/2884] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:12:44.873511 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992350e3-0bf9-4b8f-9981-5ca720627ebe None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:44.884370 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992350e3-0bf9-4b8f-9981-5ca720627ebe None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:44.889034 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:44.889630 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:44.889903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:44.890171 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:44.890812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:12:44.891109 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:44.892731 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:44.904441 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1445/2885] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:12:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:12:44.908665 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926201a9-f50f-456b-aaf3-48d7ecb38d90 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:44.918844 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926201a9-f50f-456b-aaf3-48d7ecb38d90 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:44.923426 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:44.924037 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:44.924314 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:44.925219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:44.926080 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:44.927958 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:44.954034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1441/2886] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:12:44 2022] POST /identity/v3/projects => generated 367 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:44.958514 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:44.968262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f37199-465e-41e7-9a49-20ff88ba114c admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:45.023803 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:45.024419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:45.024703 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:45.024966 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:45.025583 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:45.027462 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:45.048729 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1446/2887] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:12:44 2022] POST /identity/v3/users => generated 284 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:12:45.053446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:45.063466 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b4f53b-acad-46e1-ab37-ebddd730af98 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:45.068180 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:45.068759 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:45.069058 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:45.069326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:45.069950 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:45.071544 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:45.081407 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1442/2888] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:45 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:45.085217 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:45.094899 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a87624-a2ae-4276-a9df-2484b749c692 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:45.100172 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f242cef-a890-4fae-a6f7-274d402f1bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:45.100749 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f242cef-a890-4fae-a6f7-274d402f1bbb admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:45.101025 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f242cef-a890-4fae-a6f7-274d402f1bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:45.101288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f242cef-a890-4fae-a6f7-274d402f1bbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:45.101935 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f242cef-a890-4fae-a6f7-274d402f1bbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:45.103503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f242cef-a890-4fae-a6f7-274d402f1bbb admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:45.114911 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1447/2889] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:12:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:12:45.119102 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:45.130117 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3cf07e-c752-4d60-b95e-e73fe2e0fe1a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:45.134707 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:45.135337 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:45.135614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:45.135951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] PATH_INFO: `/v3/projects/d4127c079a684029bedf32b6bd7a105e/users/1a45859b611745ed8ca9aea76848e17d/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:45.150951 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4127c079a684029bedf32b6bd7a105e, user_id=1a45859b611745ed8ca9aea76848e17d, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:45.153606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:45.165825 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1443/2890] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:12:45 2022] PUT /identity/v3/projects/d4127c079a684029bedf32b6bd7a105e/users/1a45859b611745ed8ca9aea76848e17d/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:12:45.178748 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d433a68-ee75-4de4-9604-0a160ab03fc1 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:45.179043 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d433a68-ee75-4de4-9604-0a160ab03fc1 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:45.179321 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d433a68-ee75-4de4-9604-0a160ab03fc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:45.209424 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-2d433a68-ee75-4de4-9604-0a160ab03fc1 None None] MFA Rules not processed for user `1a45859b611745ed8ca9aea76848e17d`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:12:45.210622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2d433a68-ee75-4de4-9604-0a160ab03fc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:45.276909 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1448/2891] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:12:45 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:12:45.288861 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:12:45.290571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:12:45.323033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:45.370026 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ace7aa7-7be5-4531-9a8e-6c3e43bfd2ac admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:12:45.373921 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb26331a-ff37-49a8-8f0b-c43430bde1f8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '30b10ebdc87a4747bef7fc60b3cec9c8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:12:45.374508 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26331a-ff37-49a8-8f0b-c43430bde1f8 service glance] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:12:45.374783 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26331a-ff37-49a8-8f0b-c43430bde1f8 service glance] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:12:45.375048 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26331a-ff37-49a8-8f0b-c43430bde1f8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:12:45.384052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26331a-ff37-49a8-8f0b-c43430bde1f8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:12:45.385454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26331a-ff37-49a8-8f0b-c43430bde1f8 service glance] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:12:45.394294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1444/2892] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:12:45 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:12:55.274066 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=95,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:12:55.390083 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=22 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:26.176379 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d433a68-ee75-4de4-9604-0a160ab03fc1 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:26.238813 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d433a68-ee75-4de4-9604-0a160ab03fc1 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:26.243797 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:26.244456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:26.244735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:26.245003 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] PATH_INFO: `/v3/users/1a45859b611745ed8ca9aea76848e17d` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:26.246372 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a45859b611745ed8ca9aea76848e17d)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:26.248314 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:26.293958 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a45859b611745ed8ca9aea76848e17d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.294593 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a45859b611745ed8ca9aea76848e17d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.295194 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a45859b611745ed8ca9aea76848e17d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.302046 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a45859b611745ed8ca9aea76848e17d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.313724 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a45859b611745ed8ca9aea76848e17d'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.327096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1449/2893] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:13:26 2022] DELETE /identity/v3/users/1a45859b611745ed8ca9aea76848e17d => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:13:26.331773 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:26.332315 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:26.332437 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:26.351988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:26.353606 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:26.411397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1445/2894] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:13:26.420923 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:26.422318 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:26.499639 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7103fbc2-5dfe-43ad-b7fe-942b06a28d70 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:26.503309 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e35024-dc6b-4d24-b3b3-a61f57841114 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:26.504038 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e35024-dc6b-4d24-b3b3-a61f57841114 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:26.504320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e35024-dc6b-4d24-b3b3-a61f57841114 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:26.504591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e35024-dc6b-4d24-b3b3-a61f57841114 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:26.514130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e35024-dc6b-4d24-b3b3-a61f57841114 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:26.515538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e35024-dc6b-4d24-b3b3-a61f57841114 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:26.525448 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1450/2895] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:13:26.789640 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:26.791790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:26.813582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530d26f0-0ff4-407f-a4d9-e26621b56ae7 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:26.818376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:26.819066 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:26.819275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:26.819547 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] PATH_INFO: `/v3/projects/d4127c079a684029bedf32b6bd7a105e` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:26.820789 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4127c079a684029bedf32b6bd7a105e)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:26.823505 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:26.869602 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4127c079a684029bedf32b6bd7a105e', 'request_id': 'req-8755e942-19d8-4892-a868-069025f00fcf'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.876116 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4127c079a684029bedf32b6bd7a105e', 'request_id': 'req-8755e942-19d8-4892-a868-069025f00fcf'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.881645 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4127c079a684029bedf32b6bd7a105e', 'request_id': 'req-8755e942-19d8-4892-a868-069025f00fcf'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.894219 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] The token cache is being invalidate because project d4127c079a684029bedf32b6bd7a105e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:13:26.894502 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.895120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-8755e942-19d8-4892-a868-069025f00fcf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:26.896600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1446/2896] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:13:26 2022] DELETE /identity/v3/projects/d4127c079a684029bedf32b6bd7a105e => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:13:26.901523 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59711c17-cb2a-4e4c-9bcf-5cf744099004 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:26.901827 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59711c17-cb2a-4e4c-9bcf-5cf744099004 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:26.902101 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59711c17-cb2a-4e4c-9bcf-5cf744099004 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:26.909567 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:26.924436 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-59711c17-cb2a-4e4c-9bcf-5cf744099004 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:26.925558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-59711c17-cb2a-4e4c-9bcf-5cf744099004 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:26.984155 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1451/2897] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:26.993474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094027f-d26b-4157-bdc1-63929847882c None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:26.994018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094027f-d26b-4157-bdc1-63929847882c None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:26.994359 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094027f-d26b-4157-bdc1-63929847882c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.014988 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-e094027f-d26b-4157-bdc1-63929847882c None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:27.016217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-e094027f-d26b-4157-bdc1-63929847882c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:27.024044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1447/2898] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:27.028283 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59711c17-cb2a-4e4c-9bcf-5cf744099004 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:27.037809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59711c17-cb2a-4e4c-9bcf-5cf744099004 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:27.042728 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:27.043320 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.043597 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.043870 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] PATH_INFO: `/v3/domains` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.044532 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81424) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:13:27.044830 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:27.046437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:27.058437 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1452/2899] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:13:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 16 02:13:27.062494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094027f-d26b-4157-bdc1-63929847882c None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:27.072320 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094027f-d26b-4157-bdc1-63929847882c None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:27.077234 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:27.077846 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.078124 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.078392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.079021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:27.080985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:27.108978 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1448/2900] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:13:27 2022] POST /identity/v3/projects => generated 381 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:13:27.113394 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:27.124263 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5a30e8-88ee-47ed-a850-c34e9a723887 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:27.180396 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:27.181625 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.181732 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.181952 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.182571 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:27.184496 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:27.205107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1453/2901] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:13:27 2022] POST /identity/v3/users => generated 291 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:13:27.209326 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:27.219447 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801bbf43-ac4b-461f-8016-7f791ad0b6cd admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:27.223973 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:27.224556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.224830 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.225101 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.225728 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:27.228038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:27.238275 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1449/2902] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:13:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:13:27.242158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:27.253487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e103625-adfd-4746-9933-685e5f04aafe admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:27.258093 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8be677-1463-43a6-877b-646b3a646090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:27.258969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8be677-1463-43a6-877b-646b3a646090 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.259257 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8be677-1463-43a6-877b-646b3a646090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.259529 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8be677-1463-43a6-877b-646b3a646090 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.260225 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8be677-1463-43a6-877b-646b3a646090 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:27.261814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8be677-1463-43a6-877b-646b3a646090 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:27.272405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1454/2903] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:13:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:13:27.276458 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:27.286511 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e51469c-15a8-4f9b-b8f3-4b2fb7eca1d7 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:27.291636 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:27.292306 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.292581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.292848 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] PATH_INFO: `/v3/projects/bdf1ae945ff546d9a7a99026f4f8fbcb/users/3d21bff735c34e92b09bfba1f1a76dd8/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.308061 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf1ae945ff546d9a7a99026f4f8fbcb, user_id=3d21bff735c34e92b09bfba1f1a76dd8, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:27.310712 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:27.323517 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1450/2904] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:13:27 2022] PUT /identity/v3/projects/bdf1ae945ff546d9a7a99026f4f8fbcb/users/3d21bff735c34e92b09bfba1f1a76dd8/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:13:27.333819 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9e77c7-4029-440e-8822-03d11b560b7c None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.334135 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9e77c7-4029-440e-8822-03d11b560b7c None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.334427 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9e77c7-4029-440e-8822-03d11b560b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.362634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-2d9e77c7-4029-440e-8822-03d11b560b7c None None] MFA Rules not processed for user `3d21bff735c34e92b09bfba1f1a76dd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:27.363816 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-2d9e77c7-4029-440e-8822-03d11b560b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:27.422506 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:27.428264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1455/2905] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:27 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:27.440255 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:27.442044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:27.465745 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:27.521600 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d811dfbe-8a04-4f43-9fb4-78a9b36a0fbf admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:27.525269 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:27.525851 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:27.526128 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:27.526395 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:27.534509 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:27.535912 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:27.544550 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1451/2906] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:27 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:13:28.015919 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9e77c7-4029-440e-8822-03d11b560b7c None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:28.025555 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d9e77c7-4029-440e-8822-03d11b560b7c None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:28.029016 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-face5ebe-239e-44ed-9c0c-6e534e686800 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7b63222dc2846f5a42208877b0de123', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:28.029634 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-face5ebe-239e-44ed-9c0c-6e534e686800 service cinder] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:28.029912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-face5ebe-239e-44ed-9c0c-6e534e686800 service cinder] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:28.030181 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-face5ebe-239e-44ed-9c0c-6e534e686800 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:28.031839 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-face5ebe-239e-44ed-9c0c-6e534e686800 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:28.090195 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-face5ebe-239e-44ed-9c0c-6e534e686800 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:28.091626 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-face5ebe-239e-44ed-9c0c-6e534e686800 service cinder] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:28.100591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1456/2907] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:28 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:13:33.251754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:33.312158 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7e2490-0699-4953-822d-24972c185edb service cinder] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:33.316604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:33.317217 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:33.317494 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:33.317760 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] PATH_INFO: `/v3/users/3d21bff735c34e92b09bfba1f1a76dd8` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:33.318864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d21bff735c34e92b09bfba1f1a76dd8)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:33.320827 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:33.365947 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d21bff735c34e92b09bfba1f1a76dd8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:33.366586 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d21bff735c34e92b09bfba1f1a76dd8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:33.374443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d21bff735c34e92b09bfba1f1a76dd8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:33.375068 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d21bff735c34e92b09bfba1f1a76dd8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:33.387261 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d21bff735c34e92b09bfba1f1a76dd8'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:33.400108 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1452/2908] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:13:33 2022] DELETE /identity/v3/users/3d21bff735c34e92b09bfba1f1a76dd8 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:13:33.404489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:33.404884 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:33.405020 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:33.424525 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:33.425719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:33.488667 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1457/2909] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:33.499813 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:33.501283 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:33.528581 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:33.562535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc437d4-f06e-4829-a214-b42a399966e0 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:33.566560 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7604697-656b-41a7-b19c-25479a9e77c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:33.567141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7604697-656b-41a7-b19c-25479a9e77c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:33.567535 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7604697-656b-41a7-b19c-25479a9e77c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:33.567808 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7604697-656b-41a7-b19c-25479a9e77c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:33.577241 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7604697-656b-41a7-b19c-25479a9e77c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:33.578788 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7604697-656b-41a7-b19c-25479a9e77c5 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:33.589096 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1453/2910] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:13:33.904584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:33.906079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:33.918476 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fdf01c-59c3-4013-af8c-518987089266 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:33.924460 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:33.925160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:33.925513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:33.925922 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] PATH_INFO: `/v3/projects/bdf1ae945ff546d9a7a99026f4f8fbcb` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:33.927238 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdf1ae945ff546d9a7a99026f4f8fbcb)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:33.930443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:33.979821 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdf1ae945ff546d9a7a99026f4f8fbcb', 'request_id': 'req-4cffb204-eb71-43f6-9f55-758bd79d8550'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:33.986091 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdf1ae945ff546d9a7a99026f4f8fbcb', 'request_id': 'req-4cffb204-eb71-43f6-9f55-758bd79d8550'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:33.991591 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdf1ae945ff546d9a7a99026f4f8fbcb', 'request_id': 'req-4cffb204-eb71-43f6-9f55-758bd79d8550'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:34.005221 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] The token cache is being invalidate because project bdf1ae945ff546d9a7a99026f4f8fbcb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:13:34.005538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:34.006156 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-4cffb204-eb71-43f6-9f55-758bd79d8550 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:13:34.007623 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1458/2911] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:13:33 2022] DELETE /identity/v3/projects/bdf1ae945ff546d9a7a99026f4f8fbcb => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:13:34.012889 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9afae-893c-41c5-b7e7-201f7fdf9f7b None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.013172 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9afae-893c-41c5-b7e7-201f7fdf9f7b None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.013443 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9afae-893c-41c5-b7e7-201f7fdf9f7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.032679 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-aeb9afae-893c-41c5-b7e7-201f7fdf9f7b None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:34.033919 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-aeb9afae-893c-41c5-b7e7-201f7fdf9f7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:34.092985 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1454/2912] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:13:34.100871 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034da2c-f4a7-484b-a8c0-18770033109e None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.101160 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034da2c-f4a7-484b-a8c0-18770033109e None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.101433 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6034da2c-f4a7-484b-a8c0-18770033109e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.121912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-6034da2c-f4a7-484b-a8c0-18770033109e None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:34.123111 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6034da2c-f4a7-484b-a8c0-18770033109e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:34.130826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1459/2913] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:34.135667 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9afae-893c-41c5-b7e7-201f7fdf9f7b None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:34.145246 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9afae-893c-41c5-b7e7-201f7fdf9f7b None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:34.150007 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:34.150587 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.150862 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.151127 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] PATH_INFO: `/v3/domains` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.151754 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81423) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 16 02:13:34.152088 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:34.153735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:34.165594 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1455/2914] 10.222.0.42 () {58 vars in 1153 bytes} [Fri Sep 16 02:13:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 16 02:13:34.169799 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6034da2c-f4a7-484b-a8c0-18770033109e None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:34.178875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6034da2c-f4a7-484b-a8c0-18770033109e None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:34.184107 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:34.184692 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.184969 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.185237 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.185862 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:34.187737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:34.199024 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:34.216832 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1460/2915] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:13:34 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:13:34.221033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:34.231369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec5ce72-6256-4b56-a6e2-d9dc8429bf5e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:34.293169 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:34.293910 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.294038 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.294192 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] PATH_INFO: `/v3/users` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.294821 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:34.296752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:34.317339 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1456/2916] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:13:34 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:13:34.322192 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:34.332252 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31d6875-4ede-4321-8a19-9a8aeecc93b6 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:34.337492 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:34.338130 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.338409 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.338675 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] PATH_INFO: `/v3/roles` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.339298 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:34.340886 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:34.350653 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1461/2917] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 16 02:13:34.355263 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:34.365248 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fccb6606-ed64-4a1b-a5ea-06adb2e32b55 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:34.370140 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fe0144-097d-4821-85c8-7e3e31488a1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:34.370718 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe0144-097d-4821-85c8-7e3e31488a1c admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.371567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe0144-097d-4821-85c8-7e3e31488a1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.371836 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe0144-097d-4821-85c8-7e3e31488a1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.372749 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe0144-097d-4821-85c8-7e3e31488a1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:34.374318 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe0144-097d-4821-85c8-7e3e31488a1c admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:34.385040 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1457/2918] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:13:34.388961 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:34.399500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf6408e-6572-43db-8ecd-1ba3b2fe1b36 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:34.405029 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:34.405663 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.405939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.406207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] PATH_INFO: `/v3/projects/f1bf0e348b4944e8907b80446a12a6ac/users/c3333a4ba4204c80ac03c960bfb77ee3/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.421657 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1bf0e348b4944e8907b80446a12a6ac, user_id=c3333a4ba4204c80ac03c960bfb77ee3, role_id=f17ffc45933f41aa9bb0c2f5cd6c5c85)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:34.424296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:34.437058 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1462/2919] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:13:34 2022] PUT /identity/v3/projects/f1bf0e348b4944e8907b80446a12a6ac/users/c3333a4ba4204c80ac03c960bfb77ee3/roles/f17ffc45933f41aa9bb0c2f5cd6c5c85 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:13:34.442023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7c869e-ae7f-41fa-b7b3-960701049bd1 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.442304 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7c869e-ae7f-41fa-b7b3-960701049bd1 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.442571 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7c869e-ae7f-41fa-b7b3-960701049bd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.462480 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-1f7c869e-ae7f-41fa-b7b3-960701049bd1 None None] MFA Rules not processed for user `a00246e1b29a4746b4e48d4dc1f5618c`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:34.463726 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-1f7c869e-ae7f-41fa-b7b3-960701049bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:34.489106 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:34.523683 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1458/2920] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:34.535989 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:34.537659 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:34.594931 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc87ede-f604-4d20-804b-6c7d2eac8f99 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:34.598560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ab082f-1d77-48b0-a697-da832c0478cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:34.599158 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ab082f-1d77-48b0-a697-da832c0478cf service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:34.599438 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ab082f-1d77-48b0-a697-da832c0478cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:34.599712 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ab082f-1d77-48b0-a697-da832c0478cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:34.608702 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ab082f-1d77-48b0-a697-da832c0478cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:34.610117 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ab082f-1d77-48b0-a697-da832c0478cf service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:34.619443 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1463/2921] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:13:38.182081 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:38.182614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:38.182614 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:38.208179 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.auth.core [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] MFA Rules not processed for user `772659a1932d45dea2c5120c4497314b`. Rule list: `[]` (Enabled: `True`). {{(pid=81423) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:38.209564 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:38.288033 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1459/2922] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:38 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 16 02:13:38.685557 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235525f-e4ee-473b-9393-ea93cefd34e0 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:38.686048 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235525f-e4ee-473b-9393-ea93cefd34e0 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:38.686163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4235525f-e4ee-473b-9393-ea93cefd34e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:38.724538 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-4235525f-e4ee-473b-9393-ea93cefd34e0 None None] MFA Rules not processed for user `c3333a4ba4204c80ac03c960bfb77ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:38.726077 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-4235525f-e4ee-473b-9393-ea93cefd34e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:38.745252 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:38.746957 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:38.761429 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d0699b-673a-4301-b7fa-618325a00146 None None] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:38.765302 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:38.766082 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:38.766360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:38.767619 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:38.769454 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:38.781877 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:38.783334 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:38.793532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1460/2923] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:38 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 02:13:38.806714 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1464/2924] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:38 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:38.811520 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:38.823673 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bb757-ab11-4d48-a8f4-6152902f112c service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:38.828360 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:38.828966 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] REQUEST_METHOD: `POST` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:38.829244 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:38.829510 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] PATH_INFO: `/v3/projects` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:38.830137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:38.832073 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:38.867990 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1461/2925] 10.222.0.42 () {60 vars in 1153 bytes} [Fri Sep 16 02:13:38 2022] POST /identity/v3/projects => generated 371 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 16 02:13:38.873787 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4235525f-e4ee-473b-9393-ea93cefd34e0 None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:38.884956 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4235525f-e4ee-473b-9393-ea93cefd34e0 None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:38.943325 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:38.958920 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:38.959717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:38.960110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:38.960455 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] PATH_INFO: `/v3/users` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:38.961092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:38.963915 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:38.987872 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1465/2926] 10.222.0.42 () {60 vars in 1144 bytes} [Fri Sep 16 02:13:38 2022] POST /identity/v3/users => generated 292 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 16 02:13:38.991864 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:39.004869 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd74098-0985-430e-aa26-f7730724746a admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:39.009997 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:39.010604 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:39.010945 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:39.011137 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:39.011778 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:39.013419 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:39.028353 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1462/2927] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:13:38 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:13:39.029939 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:39.041796 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f249c0-3b4a-4a78-81d6-7fbdd4e19757 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:39.046521 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:39.047209 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:39.047779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:39.047779 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] PATH_INFO: `/v3/projects/fc722981da144745ac23c631eb71ba4c/users/067083cacad34dc799ed1c0c82efc3b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:39.065975 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc722981da144745ac23c631eb71ba4c, user_id=067083cacad34dc799ed1c0c82efc3b9, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:39.069316 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:39.092769 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1466/2928] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:13:39 2022] PUT /identity/v3/projects/fc722981da144745ac23c631eb71ba4c/users/067083cacad34dc799ed1c0c82efc3b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:13:39.093032 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:39.105312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af137323-c830-4cdf-bff8-1dd406257f9e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:39.174383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:39.175025 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:39.175321 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:39.176222 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:39.176448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:39.178027 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:39.191133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1463/2929] 10.222.0.42 () {58 vars in 1122 bytes} [Fri Sep 16 02:13:39 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 16 02:13:39.196393 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:39.208718 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ad31b2-2142-4202-a141-3420c6b7f633 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:39.215332 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422d6a17-428c-449a-93bf-f78620cb77d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:39.216855 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422d6a17-428c-449a-93bf-f78620cb77d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:39.217296 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422d6a17-428c-449a-93bf-f78620cb77d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:39.217860 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422d6a17-428c-449a-93bf-f78620cb77d2 admin admin] PATH_INFO: `/v3/projects/fc722981da144745ac23c631eb71ba4c/users/067083cacad34dc799ed1c0c82efc3b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:39.221342 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422d6a17-428c-449a-93bf-f78620cb77d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc722981da144745ac23c631eb71ba4c, user_id=067083cacad34dc799ed1c0c82efc3b9, role_id=ea4e2ba4a97343b494b1995be6f4e1dc)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:39.224983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422d6a17-428c-449a-93bf-f78620cb77d2 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:39.245096 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1467/2930] 10.222.0.42 () {60 vars in 1483 bytes} [Fri Sep 16 02:13:39 2022] PUT /identity/v3/projects/fc722981da144745ac23c631eb71ba4c/users/067083cacad34dc799ed1c0c82efc3b9/roles/ea4e2ba4a97343b494b1995be6f4e1dc => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:13:39.335448 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:39.338676 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:39.373556 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:39.425397 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32821287-c14a-48c9-bba8-96219af09a77 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:39.500369 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:39.501168 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:39.501532 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:39.501820 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:39.503907 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:39.517482 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:39.519403 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:39.531323 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1464/2931] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:13:44.779794 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72698a27-f08e-4df5-b3c8-eaf328197932 None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:44.780303 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72698a27-f08e-4df5-b3c8-eaf328197932 None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:44.780584 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72698a27-f08e-4df5-b3c8-eaf328197932 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:44.827605 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-72698a27-f08e-4df5-b3c8-eaf328197932 None None] MFA Rules not processed for user `067083cacad34dc799ed1c0c82efc3b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:44.829098 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-72698a27-f08e-4df5-b3c8-eaf328197932 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:44.928211 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1468/2932] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:44 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:44.943898 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:44.945647 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:45.091595 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9547a6f-d344-4fe1-a76b-7b7f5653276a service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:45.098513 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:45.099204 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:45.099562 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:45.100052 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:45.143561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=100 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:45.199478 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:45.201567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:45.214336 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1465/2933] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:44 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:13:46.227826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e101c67-16a4-4d59-a8d9-817f5e36456f None None] REQUEST_METHOD: `POST` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:46.227826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e101c67-16a4-4d59-a8d9-817f5e36456f None None] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:46.227826 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e101c67-16a4-4d59-a8d9-817f5e36456f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:46.250500 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.auth.core [None req-3e101c67-16a4-4d59-a8d9-817f5e36456f None None] MFA Rules not processed for user `c3333a4ba4204c80ac03c960bfb77ee3`. Rule list: `[]` (Enabled: `True`). {{(pid=81424) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 16 02:13:46.251791 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-3e101c67-16a4-4d59-a8d9-817f5e36456f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:46.259782 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1469/2934] 10.222.0.42 () {58 vars in 958 bytes} [Fri Sep 16 02:13:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 16 02:13:55.207725 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=14 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:56.244148 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=59,keystone:INSERT=5 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:13:59.600313 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:13:59.658995 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6ee3f-b764-4a7f-a976-9238b185561e service nova] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:13:59.662548 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:13:59.663148 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:13:59.663422 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:13:59.663688 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:13:59.665392 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:13:59.730617 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:13:59.732071 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:13:59.741325 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1466/2935] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:13:59 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:14:09.247490 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e101c67-16a4-4d59-a8d9-817f5e36456f None None] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:14:09.257328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e101c67-16a4-4d59-a8d9-817f5e36456f None None] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:14:09.261546 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:14:09.262140 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:14:09.262418 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:14:09.262686 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:14:09.264459 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:14:09.273888 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:14:09.275417 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:14:09.283503 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1470/2936] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:14:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:14:09.737114 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=50 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:14:19.279873 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:14:29.946691 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:14:29.955285 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0753c4c-e3bb-4a37-b1a0-2e4188aaa524 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:14:29.958875 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:14:29.959457 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:14:29.959733 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:14:29.960034 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:14:29.969378 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:14:29.970790 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:14:29.980764 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1467/2937] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:14:29 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:14:31.729290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:14:31.730875 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:14:31.746962 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ee6f4c-eb03-449c-bb26-f8253537b54b service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:14:31.752419 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:14:31.753009 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:14:31.753286 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:14:31.753549 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:14:31.755223 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:14:31.768163 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:14:31.769717 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:14:31.781516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1471/2938] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:14:31 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 16 02:14:39.975899 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:14:41.777280 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=6 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:15:12.402474 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:15:12.410996 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f35efab-c7af-4e79-9ccb-8edf68b208c3 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:15:12.414538 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:15:12.415113 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:15:12.416133 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:15:12.416625 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:15:12.418416 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:15:12.428165 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:15:12.429583 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:15:12.437702 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1468/2939] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:15:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:15:22.434049 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:15:51.173405 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:15:51.185838 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ab5e49-a7e3-48e4-84a6-2e72481b1893 service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:15:51.189497 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:15:51.190092 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:15:51.190366 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:15:51.190627 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:15:51.192912 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:15:51.208709 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:15:51.210124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:15:51.222235 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1472/2940] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:15:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:15:52.412010 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:15:52.421332 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11df7449-7acd-43fd-9f7f-2cb8a94630c8 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:15:52.424873 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:15:52.425446 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:15:52.425719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:15:52.425983 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:15:52.427719 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:15:52.443067 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:15:52.445074 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:15:52.456615 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1469/2941] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:15:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:15:53.528588 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:15:53.539736 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebae55d-81d2-4055-a6d3-78889ecfd915 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:15:53.545374 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:15:53.545970 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:15:53.546243 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:15:53.546516 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:15:53.548429 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:15:53.561395 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:15:53.562976 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:15:53.574573 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1473/2942] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:15:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:16:02.451476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=6 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:16:03.400752 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:16:03.409735 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047f3922-0a1e-44f5-a7c3-e1a497add21c service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:16:03.413280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:16:03.413863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:16:03.414141 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:16:03.414408 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:16:03.416259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:16:03.478089 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:16:03.479514 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:16:03.489004 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1470/2943] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:16:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:16:03.569124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=12 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:16:09.496682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:16:09.498264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:16:09.513719 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d9e8ea-9ff0-487c-a43a-56aba4e06a4f service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:16:09.518414 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:16:09.519089 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:16:09.519367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:16:09.519633 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] PATH_INFO: `/v3/users/772659a1932d45dea2c5120c4497314b` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:16:09.520780 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=772659a1932d45dea2c5120c4497314b)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:16:09.522672 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:16:09.569317 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '772659a1932d45dea2c5120c4497314b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.569950 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '772659a1932d45dea2c5120c4497314b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.570550 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '772659a1932d45dea2c5120c4497314b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.577844 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '772659a1932d45dea2c5120c4497314b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.590963 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '772659a1932d45dea2c5120c4497314b'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.603654 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1474/2944] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:16:09 2022] DELETE /identity/v3/users/772659a1932d45dea2c5120c4497314b => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:16:09.775023 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:16:09.784891 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dd6a9c-9376-4927-91e4-d924891997a0 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:16:09.845019 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:16:09.845638 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:16:09.845917 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:16:09.846185 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] PATH_INFO: `/v3/projects/9cfcd2376b1c4398befa49b423aaae26` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:16:09.847280 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cfcd2376b1c4398befa49b423aaae26)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:16:09.849198 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:16:09.896206 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cfcd2376b1c4398befa49b423aaae26', 'request_id': 'req-59671275-7470-4b3e-9475-fb8cb87f6988'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.902312 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cfcd2376b1c4398befa49b423aaae26', 'request_id': 'req-59671275-7470-4b3e-9475-fb8cb87f6988'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.908018 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cfcd2376b1c4398befa49b423aaae26', 'request_id': 'req-59671275-7470-4b3e-9475-fb8cb87f6988'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.920620 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] The token cache is being invalidate because project 9cfcd2376b1c4398befa49b423aaae26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:16:09.920901 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.921515 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:09.923212 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1471/2945] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:16:09 2022] DELETE /identity/v3/projects/9cfcd2376b1c4398befa49b423aaae26 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:16:09.926974 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:16:09.928382 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:16:09.941881 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcd37e-ed79-4237-909b-e09b35087f72 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:16:09.997965 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:16:09.998513 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:16:09.998903 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:16:09.999183 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] PATH_INFO: `/v3/users/1d0974ede08a4e8fa35ef2e72a63833a` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:16:10.000331 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d0974ede08a4e8fa35ef2e72a63833a)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:16:10.002228 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:16:10.046608 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d0974ede08a4e8fa35ef2e72a63833a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.047219 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d0974ede08a4e8fa35ef2e72a63833a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.047814 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d0974ede08a4e8fa35ef2e72a63833a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.054720 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d0974ede08a4e8fa35ef2e72a63833a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.067262 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d0974ede08a4e8fa35ef2e72a63833a'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.068896 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:16:10.082929 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1475/2946] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:16:09 2022] DELETE /identity/v3/users/1d0974ede08a4e8fa35ef2e72a63833a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:16:10.250090 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:16:10.251468 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:16:10.269989 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59671275-7470-4b3e-9475-fb8cb87f6988 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:16:10.291000 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:16:10.332984 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:16:10.333616 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:16:10.333894 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:16:10.334276 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] PATH_INFO: `/v3/projects/ac353b06b5524c06b9336411462ae8ad` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:16:10.335265 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac353b06b5524c06b9336411462ae8ad)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:16:10.337259 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:16:10.385296 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac353b06b5524c06b9336411462ae8ad', 'request_id': 'req-6e556691-449d-470a-a6c7-5a1c535ba144'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.391574 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac353b06b5524c06b9336411462ae8ad', 'request_id': 'req-6e556691-449d-470a-a6c7-5a1c535ba144'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.397044 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac353b06b5524c06b9336411462ae8ad', 'request_id': 'req-6e556691-449d-470a-a6c7-5a1c535ba144'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.411186 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] The token cache is being invalidate because project ac353b06b5524c06b9336411462ae8ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81423) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:16:10.411476 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.412125 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:16:10.413601 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1472/2947] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:16:10 2022] DELETE /identity/v3/projects/ac353b06b5524c06b9336411462ae8ad => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 16 02:16:20.080147 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:INSERT=1,keystone:DELETE=1,keystone:SELECT=2 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:16:20.407627 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:17:00.042711 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:17:00.044767 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:00.102489 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5218dc-fbe5-417b-8e71-4d0f4ca539be admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:17:00.151142 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:17:00.151809 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:17:00.152114 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:17:00.152380 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:17:00.154689 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:00.165197 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:17:00.166300 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:17:00.180079 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1476/2948] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:17:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:17:00.264914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:17:00.266383 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:00.280063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e556691-449d-470a-a6c7-5a1c535ba144 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:17:00.284021 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:17:00.285228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:17:00.285228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:17:00.285228 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:17:00.286859 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:00.362639 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:17:00.364120 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:17:00.376047 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1473/2949] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:17:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:17:01.552120 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:17:01.554812 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:01.572364 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c9807f-9a12-413b-9bcb-30af928c8491 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:17:01.578985 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '603c81d781754201a5ac5b7d9b84fab6', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:17:01.579800 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:17:01.580282 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:17:01.580737 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:17:01.582456 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:01.598425 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:17:01.600450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:17:01.612959 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1477/2950] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:17:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:17:10.370661 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=34 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:17:11.608930 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=52 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:17:22.705460 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:17:22.706934 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:22.776295 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce45e688-c744-475c-9eb7-99ed25b4cadc service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:17:22.781567 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:17:22.782184 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:17:22.782455 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:17:22.782741 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:17:22.784582 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:17:22.796697 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:17:22.798308 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:17:22.809348 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1474/2951] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:17:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:17:32.804341 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=32 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:18:45.701981 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:18:45.711592 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d600c8-b588-4f3d-98fe-44acc277dfbb service nova] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:18:45.716145 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:18:45.716735 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:18:45.717010 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:18:45.717279 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:18:45.718983 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:18:45.778242 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:18:45.779665 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:18:45.788407 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1478/2952] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:18:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:18:47.555294 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:18:47.566243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7f8dfd-52db-46c6-88f9-607c04512c1d service placement] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:18:47.569822 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a343a260fa2415e84caf959933a6c13', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:18:47.570406 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:18:47.570686 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:18:47.570961 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:18:47.572670 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:18:47.635338 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:18:47.637672 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:18:47.647208 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1475/2953] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:18:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:18:50.703055 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:18:50.705234 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:18:50.728833 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738df91c-b54e-4923-a7d5-f591f06f4aa7 service neutron] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:18:50.735060 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:18:50.736288 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] REQUEST_METHOD: `GET` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:18:50.736742 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:18:50.737207 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:18:50.739454 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:18:50.755849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:18:50.758367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:18:50.773580 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1479/2954] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:18:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 71 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:18:53.651063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:18:53.653055 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:18:53.670500 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f8dd2a-f8e9-4a83-887d-1d9923b4bb68 service neutron] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:18:53.675144 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:18:53.675771 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:18:53.676086 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:18:53.676355 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] PATH_INFO: `/v3/users/c3333a4ba4204c80ac03c960bfb77ee3` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:18:53.677481 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3333a4ba4204c80ac03c960bfb77ee3)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:18:53.679387 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:18:53.727390 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3333a4ba4204c80ac03c960bfb77ee3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:53.728142 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3333a4ba4204c80ac03c960bfb77ee3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:53.735022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3333a4ba4204c80ac03c960bfb77ee3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:53.735649 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3333a4ba4204c80ac03c960bfb77ee3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:53.749863 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3333a4ba4204c80ac03c960bfb77ee3'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:53.763340 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1476/2955] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:18:53 2022] DELETE /identity/v3/users/c3333a4ba4204c80ac03c960bfb77ee3 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:18:53.940264 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:18:53.951781 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1fd08-2109-4c83-bd7b-8ef236f0d59b service placement] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:18:54.008849 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:18:54.009477 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:18:54.009757 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:18:54.010028 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] PATH_INFO: `/v3/projects/f1bf0e348b4944e8907b80446a12a6ac` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:18:54.011715 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1bf0e348b4944e8907b80446a12a6ac)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:18:54.013723 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:18:54.060450 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1bf0e348b4944e8907b80446a12a6ac', 'request_id': 'req-352370f8-ab35-4778-bc0a-1a8902fbab15'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.066622 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1bf0e348b4944e8907b80446a12a6ac', 'request_id': 'req-352370f8-ab35-4778-bc0a-1a8902fbab15'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.072124 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1bf0e348b4944e8907b80446a12a6ac', 'request_id': 'req-352370f8-ab35-4778-bc0a-1a8902fbab15'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.084465 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] The token cache is being invalidate because project f1bf0e348b4944e8907b80446a12a6ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:18:54.084753 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.085367 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.086820 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1480/2956] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:18:53 2022] DELETE /identity/v3/projects/f1bf0e348b4944e8907b80446a12a6ac => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:18:54.090914 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:18:54.092376 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:18:54.105415 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe08e7f1-8d4d-467c-b38b-56419df81420 admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:18:54.161706 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:18:54.162243 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:18:54.162524 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:18:54.162795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] PATH_INFO: `/v3/users/067083cacad34dc799ed1c0c82efc3b9` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:18:54.163916 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] RBAC: Authorizing `identity:delete_user(user_id=067083cacad34dc799ed1c0c82efc3b9)` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:18:54.165849 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:18:54.171772 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:18:54.212430 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '067083cacad34dc799ed1c0c82efc3b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.213063 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '067083cacad34dc799ed1c0c82efc3b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.220953 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '067083cacad34dc799ed1c0c82efc3b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.221575 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '067083cacad34dc799ed1c0c82efc3b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.234373 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.notifications [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '067083cacad34dc799ed1c0c82efc3b9'} {{(pid=81423) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.247561 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1477/2957] 10.222.0.42 () {58 vars in 1224 bytes} [Fri Sep 16 02:18:54 2022] DELETE /identity/v3/users/067083cacad34dc799ed1c0c82efc3b9 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:18:54.423052 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Authenticating user token {{(pid=81424) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:18:54.425099 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.fernet_utils [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81424) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:18:54.446566 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352370f8-ab35-4778-bc0a-1a8902fbab15 admin admin] Validating token access rules against request {{(pid=81424) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:18:54.458682 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:18:54.509345 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a00246e1b29a4746b4e48d4dc1f5618c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efec9f6da9c342038dd7466c677c23ad', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81424) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:18:54.510159 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:18:54.510290 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:18:54.510560 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] PATH_INFO: `/v3/projects/fc722981da144745ac23c631eb71ba4c` {{(pid=81424) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:18:54.512487 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc722981da144745ac23c631eb71ba4c)` {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:18:54.514457 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] RBAC: Authorization granted {{(pid=81424) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:18:54.560110 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc722981da144745ac23c631eb71ba4c', 'request_id': 'req-3ca839f7-ec62-4227-a445-4c0fc81109ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.566328 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc722981da144745ac23c631eb71ba4c', 'request_id': 'req-3ca839f7-ec62-4227-a445-4c0fc81109ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.571828 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc722981da144745ac23c631eb71ba4c', 'request_id': 'req-3ca839f7-ec62-4227-a445-4c0fc81109ff'} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.584270 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] The token cache is being invalidate because project fc722981da144745ac23c631eb71ba4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81424) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 16 02:18:54.584558 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.585178 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG keystone.notifications [None req-3ca839f7-ec62-4227-a445-4c0fc81109ff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81424) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 16 02:18:54.586651 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: [pid: 81424|app: 0|req: 1481/2958] 10.222.0.42 () {58 vars in 1233 bytes} [Fri Sep 16 02:18:54 2022] DELETE /identity/v3/projects/fc722981da144745ac23c631eb71ba4c => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 16 02:19:04.243795 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:19:04.581401 dv-cinder-849143-2-iscsi devstack@keystone.service[81424]: DEBUG dbcounter [-] [81424] Writing DB stats keystone:SELECT=44,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81424) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 16 02:22:24.669384 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Authenticating user token {{(pid=81423) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 16 02:22:24.670761 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:22:24.809105 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a8b8c5-fead-4b13-b5a2-df17eac9961e admin admin] Validating token access rules against request {{(pid=81423) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 16 02:22:24.868231 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59b6b39-cda0-4719-9e8b-ccd24ee1838b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b4311561ef9743988bdf64a039c0628f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '967b318932ae4b66bac5ab6a6be87e75', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81423) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 16 02:22:24.868748 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b6b39-cda0-4719-9e8b-ccd24ee1838b service placement] REQUEST_METHOD: `GET` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 16 02:22:24.869024 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b6b39-cda0-4719-9e8b-ccd24ee1838b service placement] SCRIPT_NAME: `/identity` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 16 02:22:24.869289 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b6b39-cda0-4719-9e8b-ccd24ee1838b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81423) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 16 02:22:24.871576 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.fernet_utils [None req-f59b6b39-cda0-4719-9e8b-ccd24ee1838b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81423) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 16 02:22:24.882442 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b6b39-cda0-4719-9e8b-ccd24ee1838b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 16 02:22:24.884022 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b6b39-cda0-4719-9e8b-ccd24ee1838b service placement] RBAC: Authorization granted {{(pid=81423) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 16 02:22:24.893739 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: [pid: 81423|app: 0|req: 1478/2959] 10.222.0.42 () {60 vars in 1365 bytes} [Fri Sep 16 02:22:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 16 02:22:34.889565 dv-cinder-849143-2-iscsi devstack@keystone.service[81423]: DEBUG dbcounter [-] [81423] Writing DB stats keystone:SELECT=89 {{(pid=81423) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}